LLM API Concurrency Limits and a Circuit Breaker in Python with asyncio

Your script fires 40 requests with asyncio.gather, the provider answers about a quarter of them, and the rest come back 429. Your retry logic wakes up, resends them, and now you are sending a second wave into the same wall. The fix is not smarter backoff. It is not sending the wave in the first place. A bounded asyncio.Semaphore caps concurrency, and a small circuit breaker stops you hammering an API that is down.

This post builds both, runs them against a local fake server, and shows the attempt counts. Backoff, jitter and idempotency keys are already covered in LLM API rate limits and retries in Python; the timeout and fallback side is in LLM gateway retries, timeouts and fallbacks. This one is about the layer in front of them.

Disclosure: the AICredits link below is an affiliate link. I only mention it because the client works with any OpenAI-compatible base URL.

What do the providers say about 429s?

I read both vendors' rate-limit pages on 2026-10-23. Only these points matter here, and I am not adding anything the pages do not state.

OpenAI (rate limits guide):

• A 429 can carry a Retry-After header, which the page describes as the minimum seconds to wait. It advises adding a small random delay on top.
• If Retry-After is missing, use exponential backoff with jitter, and cap both attempts and total retry time.
• If you manage retries yourself, disable SDK retries or count them, so nested retries do not multiply requests.
• Failed requests still count toward your per-minute limit, so resending continuously makes things worse.
• Do not retry quota or billing errors that need you to act.

Anthropic (rate limits):

• A 429 may include retry-after, in seconds. Earlier retries will fail. Not every 429 carries it.
• Limits use a token bucket, so capacity refills continuously. A nominal 60 RPM may be enforced as one request per second, so short bursts can trip the limit.
• Sharp increases in usage can trigger acceleration limits. The page says to raise traffic gradually.
• Hitting the monthly spend cap also returns a 429 with type rate_limit_error and no retry-after. The page says to tell it apart using error.details.error_code, which is enforced_spend_limit_reached. Retrying it fails until access resumes.

Two practical consequences. First, a burst is a problem even when your average is fine. Second, some 429s must never be retried, so your retry code needs a way to classify them. If you want the tier numbers, see the Claude API rate limits guide.

How do you cap LLM API concurrency with asyncio.Semaphore?

Create one semaphore per API key (or per model, if limits differ), and hold it only while a request is in flight. The detail that matters: release it before you sleep between retries. If a task holds a slot while backing off, a few unlucky requests can starve everyone else.

async with self.sem:              # slot held only during the HTTP call
    resp = await self.http.post("/v1/chat", json=payload)
# slot released here, then we sleep and retry
await asyncio.sleep(self.delay(attempt, resp))

What number goes in the semaphore? Providers limit requests and tokens per minute, not concurrency, so concurrency is a proxy. By Little's law, in-flight requests are roughly arrival rate times latency. At 300 RPM (5 per second) and 2 seconds per call you need about 10 slots to stay on the limit. If your prompts are long, the token limit may bind first and you need fewer. Start low and raise it while the 429 count stays at zero.

The client, in full

This is the whole client I tested. It uses httpx and the standard library. The jitter is full jitter (a random delay between zero and the exponential cap), and a Retry-After value, when present, is treated as a minimum with a little random time added, which matches the OpenAI advice above.

import asyncio, random, time
import httpx

RETRYABLE = {429, 500, 502, 503, 504}

class CircuitOpen(Exception):
    pass

class Breaker:
    """Open after N consecutive failures; after a cooldown let one probe through."""
    def __init__(self, threshold=5, cooldown=1.0):
        self.threshold, self.cooldown = threshold, cooldown
        self.fails, self.opened_at, self.probing = 0, None, False

    def before(self):
        if self.opened_at is None:
            return
        if time.monotonic() - self.opened_at < self.cooldown or self.probing:
            raise CircuitOpen("circuit open, failing fast")
        self.probing = True            # half-open: exactly one request goes through

    def success(self):
        self.fails, self.opened_at, self.probing = 0, None, False

    def failure(self):
        self.probing = False
        self.fails += 1
        if self.fails >= self.threshold:
            self.opened_at = time.monotonic()

class Client:
    def __init__(self, base_url, max_concurrency=4, max_attempts=4,
                 base=0.05, cap=1.0, breaker=None):
        self.http = httpx.AsyncClient(base_url=base_url, timeout=5)
        self.sem = asyncio.Semaphore(max_concurrency)
        self.max_attempts, self.base, self.cap = max_attempts, base, cap
        self.breaker, self.attempts = breaker, 0

    def delay(self, attempt, resp):
        ra = resp.headers.get("retry-after")
        if ra and ra.replace(".", "", 1).isdigit():
            return float(ra) + random.uniform(0, 0.1)
        return random.uniform(0, min(self.cap, self.base * 2 ** attempt))

    async def chat(self, payload):
        last = None
        for attempt in range(self.max_attempts):
            if self.breaker:
                self.breaker.before()
            async with self.sem:
                self.attempts += 1
                resp = await self.http.post("/v1/chat", json=payload)
            if resp.status_code not in RETRYABLE:
                if self.breaker:
                    self.breaker.success()
                resp.raise_for_status()
                return resp.json()
            last = resp
            if self.breaker:
                self.breaker.failure()
            await asyncio.sleep(self.delay(attempt, resp))
        raise RuntimeError(f"gave up after {self.max_attempts} attempts, last status {last.status_code}")

Pointing it at a real provider means changing the base URL, the path and the auth header. For an OpenAI-compatible endpoint such as AICredits (INR via UPI, from Rs 50; base URL https://api.aicredits.in/v1), that is /chat/completions plus a bearer token. I did not call any real API for this post.

What happened when I ran it?

What I stubbed: no real LLM was called. The test server is a small FastAPI app I wrote. It takes 0.1 seconds per request, allows 5 requests in flight, and returns 429 (with no Retry-After) to the sixth. It can also be switched into an outage where every request returns 500. Real providers limit per minute, so this mock is a model of the failure shape, not of their behaviour.

I ran 40 requests at once with three concurrency settings, on 2026-10-23, Python 3.13, with a fixed random seed:

no limit (40 at once)  requests=40 client_attempts=164 server_429=127 peak_inflight=5 gave_up=3 wall=1.48s
Semaphore(5)           requests=40 client_attempts=40 server_429=0 peak_inflight=5 gave_up=0 wall=0.88s
Semaphore(4)           requests=40 client_attempts=40 server_429=0 peak_inflight=4 gave_up=0 wall=1.13s

Read the first row carefully. Forty requests turned into 164 attempts, 127 of them rejected, and three requests still failed after six attempts. The unthrottled run was also slower than the semaphore run (1.48 seconds against 0.88), because the time went into backoff sleeps instead of work.

The second row is the interesting one. Setting the semaphore exactly equal to the server's limit produced zero 429s. In real life you will not know the limit this precisely, and other processes share it, so leave headroom. Semaphore(4) cost 0.25 seconds more on this toy workload and left one slot for someone else. That is the trade: a little throughput for a quiet error log.

Also notice that the semaphore does not make retries unnecessary. It made them unnecessary in this run, because the only failure was overload. A real network will still produce the odd 502.

What does a circuit breaker add?

Backoff assumes the problem is brief. When the API is down for minutes, each call burns all its retries and then fails, and a queue of callers makes the pile bigger. A breaker notices consecutive failures, opens, and fails the next calls instantly. After a cooldown it lets a single probe through. If the probe succeeds it closes again.

I switched the server into outage mode (every request returns 500) and made 20 sequential calls, each allowed 3 attempts:

breaker=False 20 calls during outage: server_500=60 failed_fast=0 failed_after_retries=20 wall=3.49s
breaker=True  20 calls during outage: server_500=5 failed_fast=19 failed_after_retries=1 wall=0.30s
after outage ends + cooldown: probe -> {'text': 'ok'}, breaker_open=False

Without the breaker the server took 60 hits and the caller waited 3.49 seconds to learn that everything failed. With it, the server saw 5 hits and the other 19 calls failed in a fraction of a second. After I ended the outage and waited past the 1-second cooldown, one probe went through, succeeded, and closed the circuit.

Why the odd "1 failed after retries"? The first call used its three attempts. The second made two more, and the fifth consecutive failure opened the circuit mid-call, so its third attempt raised CircuitOpen. Those are counted as fast failures.

Choices I made that you may want to change

• I count every retryable status, including 429, as a breaker failure. With a semaphore in place, 429s should be rare, so a burst of them is a good reason to back off. If you run near the limit by design, count only 5xx responses and timeouts.
• Never retry or count a spend-cap or quota error. Check the error body first, as the Anthropic page describes, and surface it as a configuration problem instead.
• The breaker is per client object and not thread-safe. That is fine on one asyncio loop. It is not shared across processes either.
• Fail-fast needs a plan. Return a cached answer, queue the job, or show the user a clear message. A breaker that raises into a blank page is only half the pattern.

Does this work with several workers?

No, and this is the mistake that surprises people. The semaphore lives in one event loop. Run four Uvicorn or Gunicorn workers and you have four semaphores. Either divide your target concurrency by the worker count, or move the limiting to something shared: a gateway, or a Redis counter. For team-level budgets and per-key limits, see per-key budgets and rate limits for team LLM spend.

Checklist

• One Semaphore per API key or model, sized from RPM times latency, then lowered until 429s stop.
• Release the slot before sleeping between retries.
• Honour Retry-After as a minimum, add jitter, cap attempts and total time.
• Turn off or count SDK-level retries so they do not multiply with yours.
• Classify spend-cap and quota 429s as non-retryable.
• A breaker with a threshold, a cooldown and exactly one probe.
• Log attempts per request. The ratio of attempts to requests (164 to 40 above) is the number to alert on.

What I did not verify

I did not test against OpenAI, Anthropic or any gateway. The 0.1-second latency, the 5-slot limit and the thresholds come from my test server. The vendor statements above are from the pages named, as read on 2026-10-23. Sizing numbers like "10 slots at 300 RPM" are arithmetic, not measurements.