milliseconds.ai vs LLM APIs: choose by the work

Use milliseconds.ai when your application needs a bounded decision over text: a routing label, a yes/no judgment, a rating, or an extracted value. Use a general-purpose LLM API when the job needs composition, conversation, or synthesis. Use ordinary code when the answer follows an exact rule. Combine them when a workflow contains several of these jobs.

The choice depends on what each step must deliver. A support queue needs a destination. A customer needs an explanation. Those are different requirements, even when they begin with the same message.

Structured output is available on both sides

General-purpose LLMs can return schema-constrained JSON and typed function arguments. OpenAI’s Structured Outputs guide documents schema adherence, supported schema features, and refusal handling. Its function-calling guide describes connecting model output to application tools, including strict argument schemas.

So “you need JSON” is insufficient reason to switch providers. The distinction is the scope of the service and the work you want it to own. Small language models can still be LLMs; a decision API can use generative inference internally while exposing bounded outputs.

milliseconds offers eight task-specific capabilities: yes/no, classification, classification trees, ratings, extractive answers, structured extraction, entities, and value verification. These give a decision a named contract. A general-purpose API gives you broader instructions and generation capabilities, with structured output available when needed.

A workload selection matrix

Start with the simplest option that meets the requirement.

Workload	Start with	Why	Check before shipping
Validate a known identifier format or compare an amount with a limit	Rules and ordinary code	The condition is explicit and testable	Missing values, units, and boundary cases
Route messages into defined queues or flag a specific property of text	milliseconds.ai	The result maps to a bounded decision contract	Label coverage and errors on real messages
Find a named value and highlight its location in supplied text	milliseconds.ai answer or entities	These endpoints return source spans and offsets	Wrong or missed spans, and absent answers
Write a reply, summarize several sources, or maintain a conversation	General-purpose LLM	The task requires composition or broader context	Unsupported claims and instruction handling
Route routine requests, investigate exceptions, then draft a reply	Hybrid	Each step has a different output requirement	Escalation quality and total workflow overhead

An exact invoice total already in your database does not need a model to determine whether it exceeds a spending limit. A customer’s indirect complaint may need interpretation. Keep those two operations separate.

Where milliseconds fits—and where it stops

For classification, you supply candidate labels and can describe what each means. The response includes the selected label, its normalized probability, a confidence measure, and scores for the candidates. That is useful when the application must inspect competing options before routing.

For extraction, decide whether you need a record or a location. The answer endpoint returns a span with offsets, or a null result. Structured extraction returns fields. A source span lets someone inspect the selected text; it does not establish that the source is true or that the selection answers the question correctly.

The OpenAI-compatible surface is also deliberately limited. It supports structured extraction and tool selection, but plain chat is unsupported. System and assistant messages are discarded, and json_schema.strict is ignored. Review the compatibility differences before reusing a client. Familiar request syntax does not imply equivalent behavior or strict schema enforcement.

Choose a general-purpose LLM when the requirement extends beyond those contracts. If your existing LLM already meets quality, latency, and operating-cost requirements, replacing one call with another service needs a measurable benefit.

A practical hybrid: support intake and response

Consider an illustrative support workflow. Code first validates the request and checks account access. milliseconds classifies the message into described queues. An application policy accepts suitable results and sends ambiguous cases to a general-purpose LLM with relevant account facts and policy documents. A reviewer handles unresolved cases.

flowchart TD
    accTitle: Hybrid support workflow
    accDescr: Validated messages receive a classification. Application policy routes accepted results and sends exceptions for further analysis. Unresolved cases reach a reviewer. Reply drafting follows routing.
    A[Validate request and access] --> B[Classify message]
    B --> C{Application policy}
    B -->|Request fails| H[Review queue]
    C -->|Accepted| D[Route to support queue]
    C -->|Needs more context| E[LLM with relevant facts]
    E --> F{Validate proposed route}
    F -->|Accepted| D
    F -->|Unresolved| H
    D --> G[Draft reply when needed]
    G --> I[Check reply before sending]

Routing and writing remain separate. A billing label does not authorize a refund. A selected tool does not authorize its execution. Your application still checks permissions and business rules.

milliseconds documents the cascade pattern, but escalation to a larger model is another attempt, not a guarantee of correction. Measure whether it improves the difficult cases. The draft-reply step can run for any queue that needs it, regardless of which model selected the queue.

Compare complete workflows

Evaluate rules, a direct LLM call, and the hybrid on the same held-out examples. Include ambiguous wording, missing information, multiple intents, and languages your customers actually use. Count false actions and missed actions separately.

Schema validity measures shape. It does not measure whether the answer is right; OpenAI explicitly notes that structured outputs can contain mistakes. Likewise, milliseconds’ classification probability is a normalized share among your labels, not a calibrated probability of correctness. Its confidence describes the score distribution. Neither replaces evaluation.

Choose action thresholds from labeled data using the threshold-tuning workflow. Then measure end-to-end latency, retries, escalation rate, review effort, and cost per completed task. A cascade adds a first-stage call to every request; frequent escalation can erase its benefit.

That focus on the complete request also informs Baptiste Laget’s account of rebuilding the inference stack. Treat the engineering perspective as context, and measure your own workload.

Start with one decision and an explicit failure path. For the product rationale, read why we built decision-machine-1. For another comparison within this category, see milliseconds.ai vs Jev by TypeSafe AI.