Why Every AI Provider's API Is Different (and What to Do About It)

If you've integrated a second AI model provider into an application, you've probably encountered the LLM API differences that developers rarely anticipate. The requests look almost identical, and most of them even work at first. Then a tool call returns arguments in an unexpected format, a streaming response never seems to finish, or a JSON schema you rely on is silently ignored. These differences are becoming increasingly relevant as more organisations adopt multiple providers. In one production dataset covering more than 650 organisations, the share using multiple providers increased from 23% to 40% during 2024. This article explains why LLM APIs differ, which inconsistencies create the most engineering work, and how to structure your application so that adding another provider becomes a configuration change rather than a rewrite.

Why LLM APIs Aren't Standardized

LLM APIs aren't standardized because providers are optimizing for different capabilities and releasing new features faster than a common standard can keep up. There is no universally adopted specification defining exactly how a chat API should behave, so the closest thing to a shared standard is the request format that the largest number of existing applications already support. Even when providers adopt similar interfaces, differences in their underlying implementations can still affect how requests are processed and responses are returned.

Three forces keep these APIs from converging on a single format:

• Different priorities. One provider may focus on reasoning controls, another on prompt caching, and another on multimodal input. Each designs its API around the capabilities it wants developers to use, so the request structure reflects the provider's product decisions rather than a shared specification.
• A de facto format rather than an agreed standard. OpenAI's chat completions format has become a common interface that other providers emulate because existing SDKs, frameworks, and developer tools already support it. However, copying the request structure does not guarantee identical behaviour, and providers generally maintain their own native APIs alongside their compatibility layers.
• Features evolve faster than standards. Tool calling, structured output, reasoning tokens, and server-side conversation state have all evolved rapidly. Providers introduced their own implementations to bring these capabilities to market, and developers began building applications around them before a common design could emerge. Once those implementations gained users, replacing them with a unified standard became even more difficult.

This situation is unlikely to change significantly in the near future. Even if the industry gradually converges on shared interfaces, providers will continue introducing native features and implementation-specific behaviours. Rather than waiting for the APIs to become identical, the practical solution is to isolate their differences so that most of your application can operate independently of the provider behind each request.

Where the APIs Actually Differ

LLM API differences generally fall into two categories: cosmetic differences, where providers use different names or structures for the same concept, and behavioural differences, where similar-looking requests produce different results. Cosmetic differences are usually straightforward to handle with field mappings. Behavioural differences are more difficult because they can introduce bugs that remain invisible until a particular request, model, or production condition exposes them.

The following table compares three widely used native API formats: OpenAI-style Chat Completions, Anthropic Messages, and Google's Gemini generateContent API.

Every row in this table can be mapped into a common internal representation. The challenge is understanding what each field actually means, which values are required, and how each provider behaves when a request contains unsupported parameters or encounters an unexpected condition. Those differences are what make a seemingly straightforward provider migration more complicated than changing an API endpoint.

Request Shape and the System Prompt

Differences in field names are relatively easy to handle, but system instructions introduce a more subtle compatibility problem. Some APIs represent system instructions as messages with dedicated roles, allowing them to appear within a conversation's message structure. Others accept a single top-level system field, requiring all system instructions to be supplied separately from the conversation history. When a compatibility layer translates between these formats, it may move multiple system or developer messages to the beginning of the request and combine them into a single string. As a result, instructions originally introduced partway through a conversation can end up applying from the first turn, potentially changing the model's behaviour.

Required parameters can create similar surprises. For example, an application that omits an output-token limit may work with one provider but fail validation when switched to another that requires the parameter. A compatibility layer may supply a default value, reject the request, or handle the missing field differently depending on its implementation. These are not simply differences in syntax; they can affect whether a request succeeds and how the model responds.

Tool Calls and Structured Output

Tool calling is another area where superficially similar APIs can behave differently. One provider may return tool arguments as a JSON-encoded string that your application must parse, while another returns the arguments as an object. Code that assumes every response contains a string will fail when it encounters an object, while code that expects an object may break when it receives serialized JSON. The safest approach is to normalize tool-call arguments at the API boundary so that the rest of your application always works with the same data structure.

Structured output introduces a more complicated challenge because providers can support JSON Schema without implementing exactly the same subset of the specification. A schema accepted by one API may be rejected by another or have some of its constraints ignored. For example, compatibility requirements documented for certain APIs in 2026 include explicitly declaring the type of an array's items, handling schema definitions differently across endpoints, and working around restrictions on top-level $ref references. Even when two providers both advertise structured output, you should not assume that a schema validated by one will behave identically on the other.

These differences matter because syntactically valid JSON is not necessarily valid application data. A model might return a correctly formatted JSON object that violates a required field, uses an unexpected type, or fails to satisfy a business rule. Your application should therefore validate structured responses independently rather than relying exclusively on the provider's output-format controls.

Streaming

Streaming is another source of compatibility problems. All three API families support server-sent events in relevant streaming endpoints, but they differ in their event structures and termination signals. An OpenAI-style stream typically ends with a data: [DONE] marker, while Anthropic's Messages API emits a message_stop event. Gemini's streaming behaviour differs again, with completion represented through its response and connection lifecycle rather than an OpenAI-style [DONE] marker. As explained in our streaming reference, a parser that waits exclusively for [DONE] may hang, misreport completion, or fail to release resources when used with another API format.

The differences extend beyond the final event. One provider may deliver incremental text deltas, another may emit typed events associated with individual content blocks, and another may structure streamed output around candidate content parts. Tool calls, reasoning content, and other non-text elements can also be represented differently. A reliable integration needs to normalize these events into a common internal stream while preserving the information your application actually uses.

Errors and Usage

Error responses also vary across providers. One API may return an {"error": {...}} object containing a type and code, another may wrap the error inside a top-level object with "type": "error", and a third may use a numeric code within its error payload. When a gateway passes upstream responses through without modification, provider-specific errors may reach your application in their original formats as well. Our error handling reference distinguishes between gateway-generated errors, which follow a consistent structure, and upstream errors, which retain the originating provider's response format and status code.

Usage reporting presents a related challenge. Providers differ in how they name and count input, output, reasoning, and cached tokens, making direct comparisons difficult without normalization. Some expose reasoning tokens separately, while others report additional usage details through different fields or metadata. If you want to compare cost per request, identify the most expensive features, or determine whether switching providers would reduce spending, you need a consistent internal representation of both token usage and actual charges.

What "OpenAI-Compatible" Does and Doesn't Promise

An OpenAI-compatible API accepts requests in OpenAI's chat completions format, allowing developers to reuse the official OpenAI SDK by changing the base URL and API key. This makes it convenient to experiment with different models without rewriting the request-handling code. However, compatibility at the interface level does not guarantee that every parameter is supported, that every feature behaves identically, or that a model produces the same results through a compatibility endpoint as it would through its native API.

The most concerning problem is silent parameter rejection. One major provider's compatibility documentation states that many unsupported fields are silently ignored rather than triggering validation errors. Depending on the endpoint and model, the unsupported fields may include:

• response_format, meaning a request for JSON output may not receive the expected enforcement.
• strict on tool definitions, meaning tool arguments may not be guaranteed to match the supplied schema.
• seed, presence_penalty, frequency_penalty, and logprobs.
• reasoning_effort.

The same compatibility layer also imposes restrictions such as capping temperature at 1, requiring n to equal 1, omitting the model's reasoning from the response, and not supporting that provider's prompt caching. Its documentation describes the interface primarily as a way to test and compare models, rather than as a full replacement for the native API in long-term production deployments, and recommends using the native API when the complete feature set is required.

The practical lesson is that an OpenAI-compatible endpoint can be an excellent way to evaluate a model quickly, but it is not sufficient evidence that the model is production-ready for your particular use case. If your application depends on structured output, specific sampling parameters, reasoning controls, or prompt caching, you need to verify that those features are genuinely supported rather than assuming they work because the request was accepted.

How to Check What a Model Actually Accepts

The solution is to track capabilities at the model level instead of assuming that all models exposed through the same API support the same parameters. On AVIS, the model list endpoint returns each model's supportedParameters, including capabilities such as maxTokens, temperature, tools, and thinking. It also provides parameter constraints and a strictParams flag indicating whether unsupported parameters are rejected or dropped. This information can be used to validate configurations before production requests are sent, reducing the risk of silently ignored settings.

If you call providers directly, you can implement the same approach with a capability table in your application configuration. Record which parameters each model supports, document any relevant restrictions, and verify the information against the provider's documentation when adding or upgrading a model. A startup validation check or automated integration test can then flag unsupported parameters before they affect production traffic.

How to Contain the Differences in Your Code

You cannot eliminate LLM API differences entirely, but you can prevent them from spreading throughout your application. The objective is to create a thin compatibility layer at the boundary between your application and each provider, allowing the rest of your codebase to work with one internal format. With this architecture, adding another provider primarily involves implementing or configuring an adapter rather than modifying every feature that sends or receives model requests.

Normalize at the Edges

When integrating a second provider, start by establishing a consistent internal representation and translating provider-specific formats only when requests enter or responses leave the adapter layer.

1. Define one internal message format. Choose the roles, content structures, and tool-call representation your application will use everywhere. Each provider adapter should translate between this format and the provider's native request and response structures.
2. Keep system instructions in one place. Represent system instructions consistently in your internal format, then let each adapter determine how to place them in the outgoing request. This avoids forcing the rest of your application to account for providers that handle system messages differently.
3. Parse tool arguments into objects at the boundary. Whether a provider returns a JSON-encoded string or an object, normalize the result before passing it to application code. Your internal tools should not need to know how each provider serializes its arguments.
4. Validate structured output independently. Run generated JSON through your own schema validator, even when the provider claims to support strict structured output. This catches malformed responses, unsupported schema constraints, and cases where a compatibility layer silently ignores output-format settings.
5. Normalize stop reasons. Map provider-specific completion signals into a small internal set, such as complete, length, tool_call, filtered, and error. Treat unrecognized values as unexpected states that require explicit handling rather than automatically interpreting them as successful completions.
6. Implement a stream reader for each event format. Each adapter should translate provider-specific streaming events into the same internal event types and handle the correct termination condition, whether that is a [DONE] marker, an explicit stop event, or a closed connection. This ensures that the rest of your application can process streamed responses consistently.
7. Normalize usage and errors. Record input, output, reasoning, and cached tokens under consistent internal field names. Convert provider-specific error responses into a common internal error type while preserving the original response for debugging. Retry decisions should be based on your own error categories and retry policies, not on provider-specific message strings.
8. Keep model names in configuration. Avoid hardcoding model identifiers throughout the application. Store model and provider selections in configuration so that switching models or routing requests to another provider does not require changes across the codebase.

The key is to normalize only what your application needs to handle consistently. A good abstraction layer should simplify common operations without forcing every provider's native capabilities into a lowest-common-denominator interface. When a feature genuinely depends on a provider-specific capability, expose it explicitly rather than pretending every provider supports it in the same way.

Test the Behaviour, Not Just the Request

A successful HTTP 200 response does not prove that a provider migration has worked correctly. The request may have been accepted while a critical parameter was ignored, a tool call was malformed, or the streaming parser failed to recognize the end of the response. To catch these problems, maintain a small suite of contract tests that runs against every provider and model configuration you support.

At a minimum, test a standard text completion, a forced tool call, a structured-output request validated against your schema, a streamed response that must terminate cleanly, and an intentionally invalid request that verifies your error-handling logic. Run these tests whenever you introduce a new model, add a provider, or update an API version. For models with additional capabilities, include targeted tests for the specific features your application depends on, such as reasoning controls or prompt caching.

Testing is particularly important because provider reliability can vary under different workloads and traffic conditions. In one production dataset from 2024, some providers experienced failure rates exceeding 20% during peak periods. Routing failed requests to a second provider can improve resilience, but only if the fallback path has been tested end to end, including authentication, request conversion, output validation, and error handling. Otherwise, your fallback mechanism may simply replace one failure mode with another.

Build Your Own Layer or Use a Gateway

A normalization layer is valuable regardless of whether you build your own provider integrations or use a gateway. The decision is how much of the integration and operational complexity you want to maintain yourself. A small application using one or two providers may only need a few lightweight adapters, while an application supporting many models can accumulate significant maintenance work as each provider introduces new parameters, response formats, and API versions.

A gateway does not eliminate the differences between LLM APIs, and it is worth being cautious of any platform that suggests otherwise. Its main benefit is reducing repetitive integration and operational work, including separate API keys, billing arrangements, and usage-reporting formats. You still need to account for provider-specific behaviour, particularly when your application depends on native features or parameters that are not supported through a compatibility layer.

AVIS follows a pass-through approach, exposing OpenAI-compatible, Anthropic-compatible, and Gemini-compatible interfaces behind a single API key. This allows you to continue using the SDK and interface that best fit each feature while consolidating generation records into one usage format, including token counts and costs. Model prices are passed through with a 10% platform fee on API usage, as described on our pricing page. You still need to normalize provider-specific behaviour within your application, but you can reduce the number of integrations and billing systems you need to maintain underneath it.

FAQ

Why are LLM APIs not standardized?

Each provider designs its API around its own priorities, such as reasoning controls, prompt caching, and multimodal input, and releases new features faster than a common standard can develop. OpenAI's chat format has become a de facto interface that other providers often emulate, but adopting the same request structure does not guarantee identical behaviour. Providers also maintain their native APIs, where new features and provider-specific capabilities are often introduced first.

What does OpenAI-compatible mean?

An OpenAI-compatible API accepts requests in OpenAI's chat completions format, allowing you to reuse the OpenAI SDK by changing the base URL and API key. However, compatibility does not guarantee that every parameter or feature is supported. Some compatibility layers silently ignore unsupported parameters, including settings such as response_format or strict, which can cause applications to behave differently from what their code expects.

Can I use the OpenAI SDK with other providers?

Yes, provided that the provider or gateway offers an OpenAI-compatible endpoint. This is a convenient way to experiment with different models without rewriting your request-handling code. For production applications, verify which parameters the selected model actually supports, and consider using the provider's native SDK when you depend on features that the compatible interface does not expose, such as certain prompt-caching capabilities or reasoning-related functionality.

Why does my system prompt behave differently on another provider?

Some APIs support system or developer instructions as messages within a conversation, while others accept a single top-level system field. When a compatibility layer translates between these formats, it may move multiple instruction messages to the beginning of the conversation and combine them into one string. Instructions that were originally intended to apply only from a later turn may therefore influence the entire conversation, changing the model's behaviour.

Do all LLM APIs stream the same way?

No. Although many LLM APIs use server-sent events for streaming, they differ in event formats, content structures, and termination signals. One API may end with a [DONE] marker, another may emit an explicit stop event, and another may rely on the connection lifecycle. A stream reader designed for one format can hang, terminate prematurely, or misinterpret completion when used with another provider.

Should I build my own LLM API wrapper?

You should generally maintain a thin internal abstraction so that your application can work with consistent message formats, stop reasons, usage records, and error types. Whether you should also implement every provider adapter yourself depends on the number of providers you use and the complexity of your requirements. For one or two providers, lightweight custom adapters may be sufficient. For applications using many models or switching providers frequently, a gateway can reduce the integration, billing, and usage-normalization work you would otherwise need to maintain.

Wrapping Up

LLM API differences are unlikely to disappear, and the most difficult problems are often behavioural rather than cosmetic. System prompts can be repositioned, tool arguments can arrive in different formats, structured-output constraints can be interpreted differently, streaming APIs can signal completion in incompatible ways, and unsupported parameters can be silently ignored. The most reliable way to manage these differences is to isolate provider-specific behaviour at the edges of your application, use a consistent internal format, validate model outputs independently, and maintain contract tests that verify each integration before it reaches production.

If you want to reduce the operational overhead of working with multiple providers, a gateway can consolidate API access, billing, and usage reporting without requiring you to abandon provider-specific capabilities. AVIS provides compatible interfaces for OpenAI, Anthropic, and Gemini behind a single API key, with a unified usage record for tracking tokens and costs. This lets you focus your engineering effort on the normalization and validation your application actually needs, rather than repeatedly rebuilding the same integration infrastructure for every provider.

Last updated: October 2026