Reliability

Rate limits

Each API key has its own rolling hourly request allowance. Read the response headers and pause requests when the allowance is exhausted.

Allowance and rolling window

The default limit is 1,000 requests per API key over a rolling 60-minute window. Your key may have a different allowance, so treat the response headers as authoritative.

The window is not tied to the top of the hour. A request stops counting when it becomes more than 60 minutes old. Enforcement is shared across Core API instances, including concurrent requests made with the same key.

Response headers

X-RateLimit-Limit

The key's configured maximum requests per rolling 60-minute window.

X-RateLimit-Remaining

Requests still available after the current request.

Retry-After

Seconds to wait after a 429 response. Core API currently returns 3600.

When the quota is exhausted

Core API rejects the request with 429 Too Many Requests. The rejected request does not return the requested resource. Stop sending requests with that key, wait at least the number of seconds in Retry-After, and then retry.

Retry safely

Use Retry-After for 429 responses. Use exponential backoff with jitter for other transient failures.

JavaScript
const wait = (milliseconds) =>
  new Promise((resolve) => setTimeout(resolve, milliseconds));

async function fetchWithRateLimit(url, options, maxAttempts = 3) {
  for (let attempt = 1; attempt <= maxAttempts; attempt += 1) {
    const response = await fetch(url, options);
    if (response.status !== 429) return response;

    if (attempt === maxAttempts) {
      throw new Error("API rate limit still exceeded");
    }

    const retryAfter = Number(
      response.headers.get("Retry-After") ?? "3600"
    );
    await wait(retryAfter * 1000);
  }

  throw new Error("Request was not completed");
}
Retry only safe work

GET requests are naturally repeatable. For crawl starts, reuse the exact same idempotency_key only when retrying the same intent.

Request a higher limit

Limits can be adjusted for individual keys without an API deployment. Contact [email protected] with your account, key name, expected request volume, and expected peak requests per hour. Never send your secret API key.