Allowance and rolling window
The default limit is 1,000 requests per API key over a rolling 60-minute window. Your key may have a different allowance, so treat the response headers as authoritative.
The window is not tied to the top of the hour. A request stops counting when it becomes more than 60 minutes old. Enforcement is shared across Core API instances, including concurrent requests made with the same key.
Response headers
X-RateLimit-LimitThe key's configured maximum requests per rolling 60-minute window.
X-RateLimit-RemainingRequests still available after the current request.
Retry-AfterSeconds to wait after a 429 response. Core API currently returns 3600.
When the quota is exhausted
Core API rejects the request with 429 Too Many Requests. The rejected request does not return the requested resource. Stop sending requests with that key, wait at least the number of seconds in Retry-After, and then retry.
Retry safely
Use Retry-After for 429 responses. Use exponential backoff with jitter for other transient failures.
const wait = (milliseconds) =>
new Promise((resolve) => setTimeout(resolve, milliseconds));
async function fetchWithRateLimit(url, options, maxAttempts = 3) {
for (let attempt = 1; attempt <= maxAttempts; attempt += 1) {
const response = await fetch(url, options);
if (response.status !== 429) return response;
if (attempt === maxAttempts) {
throw new Error("API rate limit still exceeded");
}
const retryAfter = Number(
response.headers.get("Retry-After") ?? "3600"
);
await wait(retryAfter * 1000);
}
throw new Error("Request was not completed");
}GET requests are naturally repeatable. For crawl starts, reuse the exact same idempotency_key only when retrying the same intent.
Request a higher limit
Limits can be adjusted for individual keys without an API deployment. Contact [email protected] with your account, key name, expected request volume, and expected peak requests per hour. Never send your secret API key.