Back to blog
Web Scraping·July 16, 2026·13 min read

Why Scrapers Still Hit HTTP 429 Errors Despite Delays

HTTP 429 errors persist despite request delays because servers enforce rate limits by identity signals (fingerprints, session state), not just timing. Learn the root causes and infrastructure solutions.

A

Arun Singh

Anakin Team

Hero graphic for the blog showing a large red 429 error code alongside a timeline of 8 spaced requests all returning 429, with a fingerprint hash chip illustrating that identity not timing is the root cause of rate limiting

Adding delays between scraper requests often fails to prevent HTTP 429 errors. Servers enforce rate limits by identity signals (browser fingerprints, session tokens and concurrent request patterns), not just by request timing. The fix is to find which identity the server is counting and pace that identity, not just the clock.

Key takeaways

  • HTTP 429 errors persist despite delays because servers group requests by fingerprint (user-agent, TLS cipher, screen resolution) and session state, not just timestamps.
  • Rate-limit algorithms like token buckets and sliding windows count requests per identifier (IP address, session token, fingerprint hash), so timing changes alone rarely help.
  • Rotating proxies while the fingerprint stays the same doesn't split the identity: servers see many IPs sharing one fingerprint hash and keep returning 429.
  • Session cookies and localStorage persist across proxy rotations unless explicitly cleared, so the server keeps grouping requests under one identity.
  • Infrastructure that keeps fingerprints consistent within a session and clears state between sessions removes most avoidable 429s without manual session management.

Why HTTP 429 errors happen even with request delays

Adding delays between requests fixes timing but not identity. Servers count requests in a window, often a sliding window over the last N seconds, but they group those requests by identifier: IP address, session token, TLS signature or browser attributes. When your scraper's identity signals cluster together, the server sees one persistent actor making many requests, even if those requests arrive seconds apart. Three hidden causes drive this clustering. Fingerprint inconsistency: requests expose User-Agent or header combinations no real browser would send, which flags the traffic as automated. Session-state leakage: cookies or tokens from one request carry into the next and link them. Concurrent bursts: parallel threads share one IP or credential, so their delays collapse into simultaneous hits.

What HTTP 429 Too Many Requests really means

HTTP 429 Too Many Requests is the server's signal that a client exceeded the allowed request rate within a time window. Websites rate-limit to protect server resources and curb abusive automation. Two models dominate: token buckets hand out a fixed number of request tokens that refill at a steady rate, while sliding windows count requests over a rolling period instead of fixed minute boundaries. When your identifier (IP address, API key or session token) exhausts its quota, the server rejects further requests with a 429, often with a Retry-After header saying when to try again.

The delay fallacy: timing vs identity

Slowing your request rate assumes the server counts only timestamps. Production rate limiters count requests per identifier, not per second. If your scraper sends ten requests over sixty seconds and all ten share one IP address and one TLS fingerprint, they land in a single bucket. The delay spreads the timeline but doesn't split the identity. Fingerprint-aware systems track User-Agent consistency, header order, cipher suite negotiation and session continuity, so adding sleep calls between identical requests changes little. That's why scrapers that rotate proxies but reuse session cookies, or vary User-Agent strings but keep identical Accept-Language headers, keep hitting 429 errors despite respecting timing intervals. Anakin's infrastructure handles this pairing for you: the URL Scraper routes requests through residential proxies in 207 countries and territories, and useBrowser sends them through a managed, fingerprint-masked browser, so a request's network and browser signals agree with each other.

Get an API key (300 free credits, no card required) and test it on a URL that keeps returning 429 for you.

Split-panel diagram contrasting what a developer configures (8 requests spaced 8 seconds apart on a 60-second timeline) with what the server actually sees: all 8 requests collapsed into one identity bucket keyed to fingerprint fp_a3f2c9, breaching the limit of 5 and returning HTTP 429

How server-side rate limiting actually works

Client-side delays address only half the problem. The server doesn't see the pauses between requests; it sees arrival timestamps and identity clusters. Most production systems layer several buckets, each tracking a different dimension of identity.

Sliding window vs token bucket algorithms

Rate-limit enforcement typically uses one of two models. A sliding window algorithm counts requests within the last N seconds and rejects once the threshold is reached. Anakin's own API works this way, per its rate limits docs: 60 requests per minute for most submit endpoints such as POST /v1/url-scraper, 10 per minute for AI evaluation, and no limit on URL Scraper and Search polling (GET /v1/url-scraper/{id}). Each endpoint is an independent bucket. The window slides continuously, so a burst at second 58 and another at second 2 of the next minute both count inside the same 60-second span.

A token bucket refills capacity at a fixed rate, five tokens per second for example. A burst consumes tokens instantly; when the bucket is empty, requests wait for the refill. This model permits short bursts but enforces a sustained average. The choice changes retry behavior: sliding windows reward spreading requests across the full period, while token buckets tolerate short bursts followed by pauses.

Per-IP vs per-session vs per-fingerprint limits

Modern servers can enforce rate limits across three independent identity layers:

  1. Per-IP bucket is the coarsest signal. Traffic spread across many addresses keeps each IP's count low, but a single residential proxy shared by many users hits the limit quickly.
  2. Per-session bucket is keyed by cookies or session tokens. Rotating proxies doesn't clear this counter if the session persists across IP changes. Dropping cookies after every request avoids building a session count, but it also gives up any logged-in state the scrape depends on.
  3. Per-fingerprint bucket groups requests into recognizable cohorts by TLS handshake patterns and HTTP/2 frame order, even when IP and session change. A default headless Puppeteer setup shares its fingerprint with many other Puppeteer instances, so one abusive actor can exhaust the bucket for the whole cohort.

Changing one dimension (switching proxies, clearing cookies or swapping user-agent strings) doesn't reset the others. The server tracks all three in parallel, and whichever bucket fills first returns the 429. So identity clustering can trip a limit long before your overall request rate gets near it.

Bar chart showing three identity buckets filling simultaneously: Per-IP at 22% because the IP is being rotated, Per-session at 58% because cookies persist across rotations, and Per-fingerprint at 106% overflowing the 80% threshold and triggering a 429 error

Three hidden causes of persistent 429 errors

Most scraping guides prescribe delays and proxy rotation as the universal 429 fix. When those fail, the root cause usually isn't volume. It's identity signals that group requests under one automated fingerprint even as IPs rotate.

Browser fingerprint inconsistency across requests

Anti-bot systems probe browser artifacts to spot automation: user-agent strings, screen resolution, timezone, installed fonts, WebGL renderer and TLS cipher-suite order. When a scraper rotates proxies but sends the same fingerprint profile, or sends attribute combinations no real browser would produce, the server groups those requests into one automated cluster and applies limits at the cluster level. The FP-Inconsistent study (ACM IMC 2025) sent half a million requests from 20 commercial bot services to a honey site protected by DataDome and BotD. The bots evaded detection 52.93% and 44.56% of the time, but their fingerprints were often internally inconsistent, and rules built on those inconsistencies cut evasion by roughly 45 to 48%.

Servers can hash request characteristics (user-agent string, TLS cipher suite, screen resolution, timezone, installed fonts, WebGL renderer) into a fingerprint that survives IP changes. If browser attributes don't change along with the proxy, the hash stays constant. Behavior counts too: research on fingerprinting AI browsing agents found that typing, scrolling and mouse patterns separated seven AI agents from humans and from each other, even where their browser fingerprints were shared.

That's the proxy rotation trap. Many IPs sharing one fingerprint hash is a pattern typical of bot infrastructure, and when the limiter buckets by fingerprint rather than IP, each new proxy connection adds to the same counter. Once the per-fingerprint threshold trips, the server returns 429 across every IP tied to that hash, and the rotation buys nothing.

Funnel diagram showing six different proxy IP addresses from the US, DE, SG, FR, BR, and ZA all converging into a single shared fingerprint hash fp_a3f2c9d1, which fills one identity bucket past its limit and returns HTTP 429

Session state leakage (cookies and localStorage persistence)

Rotating IPs does not clear cookies or localStorage unless the scraper manages the session lifecycle. When a request carries a session token or tracking cookie from a previous IP's session, the server links the new IP to the old identity and applies the cumulative count. The FP-Inconsistent study also checked consistency across time (the same attribute observed at two points), which is exactly the comparison that carried-over state makes possible.

Cookies, localStorage and IndexedDB all outlive proxy rotation, and anti-bot systems read them to recognize returning clients after the IP changes.

Session cookies carrying JWTs or authentication state stay in the scraper's context across proxy hops unless explicitly dropped, and a single login cookie can bind hundreds of requests to one identity. When a 403 or 429 points to a flagged session, clear its cookies before retrying so the next request doesn't inherit it.

Headless browser scrapers also inherit localStorage and IndexedDB entries from prior runs: device IDs, feature flags, interaction timestamps. Proxy rotation alone doesn't clear them, so each new IP carries the same client-side footprint until the browser context is torn down.

Three session-lifecycle steps limit the leakage:

  1. Clear cookies and storage before each proxy rotation to drop inherited session artifacts.
  2. Isolate browser contexts using separate profiles or container tabs so that parallel requests do not share state.
  3. Validate session expiry before retrying, so a flagged credential isn't reused after a failed request.

Concurrent request clustering and batch patterns

Servers also detect batch patterns when request timing clusters, even with random jitter added. A 2026 bot-detection measurement study summarizes earlier work that crawled the Alexa Top 10K with headless Chromium: of 291 sites that blocked the crawler, 93 (about a third) used browser fingerprinting rather than simple per-IP thresholds. When several requests from one identity (fingerprint plus session) arrive in a narrow window, batch heuristics fire regardless of per-request delays. Cap concurrency per identity and keep sessions isolated, rather than relying on spacing alone.

Diagnose which bucket you're hitting

A 429 can be tied to different identities: IP address, session, account, API key, cookie or client fingerprint. Work through these checks in order:

  1. Check for a Retry-After header. If it's present, honor it: Retry-After: 120 means wait 120 seconds.
  2. Test whether changing the IP alone stops the 429s. If it does, the limit is per-IP, and each IP needs pacing under it.
  3. If not, compare fingerprints across your requests: TLS signature, user-agent, header order. Inconsistent or default headless values point to a per-fingerprint bucket.
  4. If 429s continue, check session state. The limit may be tied to cookies or localStorage carried between requests.

Reading and acting on Retry-After headers

When a server returns HTTP 429, it often includes a Retry-After header saying how long to wait. Scrapers that ignore it and retry immediately trigger more 429s and compound the problem. Retry logic that parses and honors Retry-After breaks that loop.

Parsing Retry-After header values

The Retry-After header ships in two formats: delay-seconds (integer, e.g. 120) or an HTTP-date (absolute timestamp, e.g. Wed, 21 Oct 2026 07:28:00 GMT). Robust scrapers must handle both. Try parsing as an integer first, then fall back to an HTTP-date parser. Then add a little random jitter on top of the wait (10 to 15% works) so parallel workers don't all retry at the same instant.

import random
import time
import requests
from email.utils import parsedate_to_datetime
from datetime import datetime, timezone

def parse_retry_after(header_value: str) -> float:
    # Return seconds to wait, from either a delay-seconds or HTTP-date value.
    try:
        return max(0.0, float(int(header_value)))
    except ValueError:
        retry_at = parsedate_to_datetime(header_value)
        delta = (retry_at - datetime.now(timezone.utc)).total_seconds()
        return max(0.0, delta)

def fetch_with_backoff(url: str, max_retries: int = 5):
    for attempt in range(max_retries):
        r = requests.get(url, timeout=30)
        if r.status_code != 429:
            return r
        header = r.headers.get('Retry-After')
        # Honor Retry-After when present; otherwise back off exponentially.
        wait = parse_retry_after(header) if header else min(60, 2 ** attempt)
        # Add 10-15% jitter on top so parallel workers don't retry in lockstep.
        time.sleep(wait * random.uniform(1.10, 1.15))
    raise RuntimeError(f'Exhausted retries for {url}')

Exponential backoff implementation

When Retry-After is absent, exponential backoff is the standard fallback: start with a short delay and double it on each retry, with random jitter. The tenacity library handles this cleanly, and its wait_random_exponential applies full jitter up to a cap.

import requests
from tenacity import retry, stop_after_attempt, wait_random_exponential, retry_if_exception_type

class RateLimited(Exception):
    pass

@retry(
    retry=retry_if_exception_type(RateLimited),
    wait=wait_random_exponential(multiplier=1, max=60),
    stop=stop_after_attempt(6),
    reraise=True,
)
def scrape(url: str):
    r = requests.get(url, timeout=30)
    if r.status_code == 429:
        raise RateLimited(r.headers.get('Retry-After'))
    r.raise_for_status()
    return r.json()

Backoff alone doesn't help when every retry lands in an identity bucket that's already full. In that case, slow the identity being counted: honor Retry-After, lower concurrency per session or IP, and drop a flagged session's cookies instead of reusing them. Our 10 best fixes for HTTP 429 errors guide covers token-bucket pacing and retry budgets in more depth.

How Anakin's infrastructure reduces 429 errors

The three root causes (fingerprint inconsistency, rate-bucket accumulation and session-state leakage) add up to one infrastructure problem. Hand-built scrapers tend to patch them one at a time. Anakin's proxy routing, caching and session handling address all three at the platform level.

Proxy routing and fingerprint consistency

Anakin's URL Scraper routes requests through residential proxies across 207 countries and territories; set country to pick the exit. Adding useBrowser: true sends the request through a managed, fingerprint-masked browser engine, so you don't assemble header and fingerprint sets by hand. For interactive browser work, Browser API geo-targeting goes further: pass ?country= and the browser's timezone, locale and Accept-Language all match the proxy's country. See the submit scrape job docs for the full request surface.

curl -X POST https://api.anakin.io/v1/url-scraper \
  -H "X-API-Key: $ANAKIN_API_KEY" \
  -H "Content-Type: application/json" \
  -d '{
    "url": "https://example.com/product/123",
    "country": "us"
  }'

# 202 Accepted
# {
#   "jobId": "job_abc123xyz",
#   "status": "pending"
# }
#
# Then poll GET https://api.anakin.io/v1/url-scraper/{jobId}
# until "status" is "completed" (URL Scraper polling isn't rate-limited).

Intelligent caching for repeat requests

Repeat requests for the same URL within 24 hours are served from Anakin's cache at zero credit cost. A cached response never reaches the target server, so it adds nothing to the site's rate-limit buckets. It still counts toward your Anakin API rate limit, as the rate limits docs note. That cuts repeat load on the target without your own cache-invalidation logic.

Session-lifecycle management at the infrastructure layer

Anakin handles authenticated scraping through Browser Sessions, which isolate each user's session context and encrypt session data with AES-256-GCM. Session clearing, context isolation and credential rotation are managed at the platform layer, so session-state leakage isn't left to the developer. That prevents stale-session 429 errors without manual session teardown after every workflow run.

Diagram of Anakin infrastructure stack showing three horizontal layers (proxy routing across 207 countries and territories, caching for repeat requests, and session-lifecycle management) with incoming requests on the left flowing through and exiting as 200 OK responses on the right

Pace the identity, not just the clock

Choose exponential backoff when:

  • The 429 is infrequent and you have engineering capacity to implement retry logic with jitter.
  • Your volume stays well under the published rate limit and you can afford to slow down requests.

Choose an infrastructure service like Anakin when:

  • The limit is tied to fingerprint or session state, so IP rotation alone won't fix it.
  • You'd rather not build and maintain proxy routing, browser fingerprints and retry orchestration yourself.

Manual fingerprint management and session clearing cost nothing in infrastructure but take engineering time to build and maintain. Anakin removes that work in exchange for per-request credits. Backoff works when 429s are purely timing-based (per-IP buckets, no fingerprinting) and falls short when servers cluster requests by identity, which is why the diagnosis comes first.

The delays in your scraper were never the whole story. Servers count identities, so the reliable fix is to find the identity being counted, keep its signals consistent and pace it under the limit.

Try Anakin's URL Scraper on the URLs that keep returning 429: residential proxy routing, a managed browser engine and 24-hour caching come built in, with no fingerprint management on your side.

Frequently asked questions

Why do scrapers still get 429 errors after adding random delays between requests?

Delays address timing, not identity clustering. Servers count requests per fingerprint, IP address, session token or browser signature within a window, not per second. Requests sharing the same fingerprint accumulate in the same bucket, and once it's full, the server returns 429 no matter how they were spaced.

What's the difference between per-IP and per-session rate limits?

Per-IP limits count requests from one IP address within a time window, regardless of session. Per-session limits count requests from one session cookie or fingerprint hash, regardless of the IP address used. When servers layer both, rotating proxies solves only the per-IP constraint; per-session buckets keep filling until the session state is cleared or the session slows down.

Does rotating proxies fix HTTP 429 errors?

Not on its own when servers enforce per-fingerprint or per-session buckets. Production rate limiters count requests per identifier, not per second. Keep each session's fingerprint consistent, clear session state (cookies, localStorage) between jobs, and pace each identity under its limit.

How do you parse the Retry-After header in Python?

Check whether Retry-After is present in response.headers. Parse as an integer (delay-seconds) or as an HTTP-date string with email.utils.parsedate_to_datetime. See the parse_retry_after helper earlier in this post for a reference implementation.

What browser fingerprinting signals do servers track?

Common signals include the user-agent string, screen resolution, timezone, installed fonts, WebGL renderer and TLS cipher suite. Hashed together, they form a fingerprint that survives IP changes. Inconsistent or generic fingerprints, such as headless Chrome defaults, expose scrapers even when proxies rotate.

When does exponential backoff fail to fix 429 errors?

When every retry lands in the same identity bucket. The server sees the retries as the same automated source, so the count keeps climbing whatever the backoff interval. This happens when a scraper keeps reusing a flagged session or keeps sending from one overloaded identity; slowing that identity and clearing the flagged session's state fixes it.

How does Anakin's caching help with rate limits?

Repeat requests for the same URL within 24 hours are served from Anakin's cache at zero credit cost. The target site never sees a cached request, so its rate-limit buckets don't move, although each request still counts toward your Anakin API rate limit.