What Are Concurrency Limits, and How Do They Throttle Your App?

Your app handles ten test users without a hiccup. Then a hundred real ones arrive, nothing errors, and every generation suddenly takes four minutes. That's usually an API concurrency limit at work: a cap on how many requests your account can have running at the same moment. It's not the same as a rate limit, and it hits video and image apps far sooner than chat apps, because every slow job holds its slot for longer. This guide explains what counts against the limit, what providers do when you hit it, and how to size and queue your work so the limit stops surprising you.

What is an API concurrency limit?

An API concurrency limit is the maximum number of requests or jobs your account can have in progress at once. If the limit is 5 and five jobs are running, a sixth has to wait, or fail, until one of them finishes. It caps how much work is in flight, not how often you call.

That's the key difference from a rate limit. A rate limit counts how many requests you start per minute. A concurrency limit counts how many are still running. You can stay well under a per-minute limit and still be stuck at your concurrency limit if each request takes a long time.

Most AI platforms enforce several limits at once, and the one you hit first depends on your workload:

Providers limit concurrency because AI inference runs on scarce hardware. A single video job can take a minute or more to generate, so the number of jobs running at once is what ties up the provider's capacity. Starting limits are often small, commonly 1 or 2 concurrent jobs for a new account, with self-serve ceilings in the tens and higher limits by arrangement.

Why slow jobs hit concurrency limits first

Job duration decides how much a concurrency limit lets you do. The arithmetic is simple: jobs per minute is roughly your concurrency limit times 60, divided by how many seconds each job takes. The same limit gives very different throughput depending on the work (durations are typical examples, not benchmarks):

A limit of 5 feels generous for a chatbot and tight for a video app. If 20 users each request one video in the same minute, the last of them waits several minutes, even though the account never comes close to a rate limit. With no errors to point at, the app just looks slow.

Fan-out multiplies your concurrency

One user action often triggers several model calls. A "generate a storyboard" button might create six images at once. An agent might call a model, a tool and the model again in parallel. Count the in-flight calls per user action, not just the number of users. Some providers also count a request for four images as four slots, not one.

Retries and stuck jobs hold slots too

A retry is a new request competing for the same slots. If jobs start failing and your code retries every one of them immediately, you double the load at exactly the moment the system is short of capacity. Jobs that hang until a timeout are worse: on at least one provider, a task holds its slot until it finishes, including when it fails, so a slow failure blocks a slot for its whole life.

What counts against the limit, and what happens when you hit it

Providers agree on the idea of a concurrency limit and disagree on almost every detail of it. Before you design around one, check these six things in your provider's documentation or by testing:

Two consequences follow. First, a second API key usually doesn't raise your limit, because many providers count concurrency per account. Second, switching to another model of the same type may not help either, if the limit is shared.

A 429 isn't always a concurrency problem

429 Too Many Requests is the usual signal for "slow down", but providers use it for different limits. On AVIS, a 429 means an API key has reached the spend limit you configured for it (daily, weekly, monthly or total), per our errors reference. It doesn't mean you've run out of concurrency. Read the error body before you build retry logic around the status code, because retrying a spend-limit 429 just repeats the rejection.

A job sitting in queued isn't necessarily a concurrency problem either. Async video and image jobs on AVIS move through queued, then running or processing, then succeeded or failed, as the video generation reference shows. A short wait in queued is normal; a growing wait as traffic rises is the concurrency signal.

How to design around a concurrency limit

Treat the limit as a fixed number of slots and manage the slots yourself. Four habits cover most apps.

Size the worker pool

The number of jobs you'll have in flight on average is your arrival rate times the average job duration. If users request 2 videos a minute and each takes 90 seconds, you'll average 3 jobs in flight (2 per minute × 1.5 minutes). That's the average, not the peak: run exactly that many workers and any burst piles up in a queue that never drains. Leave headroom by keeping average usage clearly below the limit, and include retries when you count arrivals, since each retry is another attempt that needs a slot.

Then cap your workers at, or just under, your concurrency limit. Running more workers than you have slots doesn't add throughput. It just moves the queue from your side to the provider's.

Queue on your side

Put a bounded queue in front of the model calls and let a fixed pool of workers pull from it. This works whether your provider queues, throttles or rejects overflow, and it gives you control the provider can't:

• Priority lanes. Keep interactive requests in their own lane so a nightly batch job can't hold every slot while a user waits.
• A queue length limit. When the queue is full, tell the user it's busy instead of accepting work you can't finish in time. A queue with no limit just collects jobs that will be out of date by the time they run.
• Honest wait times. With a known worker count and average job time, you can show a queue position or an estimated wait.

For async jobs, store each job ID as soon as it's accepted and poll for status. At least one provider we checked states outright that status queries don't count against the limit.

Retry without making it worse

Retry only errors that can succeed later, use exponential backoff with some randomness, and cap the number of attempts. Make retries go back through your queue rather than jumping straight back to the API, so they wait for a free slot like any other job. Set a timeout for each job type based on its normal duration, so a stuck job releases its worker. Failed attempts cost more than time, too; our look at the hidden costs of running AI in production covers what retries add to the bill.

Measure it and raise it

You can't size a pool for a number you've never measured. On AVIS, Usage Query breaks usage down by API key and product type and shows concurrency over time, so you can see whether you're actually hitting a ceiling or just running slow jobs.

When you need more headroom, check how your provider raises limits. Many tie them to usage or spend. AVIS sets the concurrent-request limit per account and raises it as you use the service more or add credit, and the Enterprise team handles higher limits, per our FAQ. Because one AVIS key reaches many models through the same request shape, you can also move batch work to a faster model, which shortens each job and frees slots sooner. Just remember the account limit is shared, so a different model isn't extra capacity.

FAQ

What is a concurrency limit?

A concurrency limit is the maximum number of requests or jobs your account can have running at the same time. When you reach it, new requests wait or fail until a running one finishes. It caps work in flight, not how often you call the API.

What's the difference between a rate limit and a concurrency limit?

A rate limit counts how many requests you start in a time window, such as per minute. A concurrency limit counts how many are still running at once. Long jobs like video generation usually hit the concurrency limit long before any rate limit.

Does a queued request count toward my concurrency limit?

It depends on the provider. Some count only requests that are actively running, while others count a task from submission until it finishes, including failures. Check your provider's documentation, or test by submitting a few more jobs than your limit and watching their status.

How many workers should I run?

Start with your average arrival rate times your average job duration, which gives the average number of jobs in flight. Add headroom for bursts and retries, then cap the pool at or just under your concurrency limit. More workers than slots only moves the queue to the provider.

How do I get a higher concurrency limit?

Most providers raise limits with usage or spend, and some do it automatically as you buy more credit. For limits beyond the self-serve range, you usually need to contact the provider's sales or enterprise team. A second API key rarely helps, because limits are often counted per account.

Why is my AI generation stuck in a queue?

A short wait in a queued state is normal for async image and video jobs. If waits grow as traffic rises, your account is probably at its concurrency limit and new jobs are waiting for running ones to finish. Shorter jobs, fewer parallel calls per user action or a higher limit will reduce the wait.

Wrapping up

Concurrency limits cap how many jobs run at once, so job duration decides your real throughput: the same limit that handles 150 chat calls a minute handles about three videos. Check what your provider counts and what it does at the limit, size your workers from arrival rate times job duration, and queue on your side with priority lanes and capped retries. On AVIS, watch concurrency in Usage Query, and remember that a 429 there means a spend limit, not a full queue.

Last updated: October 2026