Tools & API docs

Everything built around the free Crawl Budget Snapshot tools: Apify actors for bulk AI-crawler audits and llms.txt checks, a browser extension for Chrome, Edge and Firefox, and the public JSON APIs behind this site's checkers.

AI Crawler & robots.txt Readiness Audit (Apify actor)

Status: live on Apify Store. Run it on Apify (aicrawlready/ai-crawler-robots-readiness-audit). Pricing: $0.01 per domain audited, pay per event.

A per-domain audit of robots.txt rules for 21 vendor-documented AI crawler tokens (GPTBot, ClaudeBot, Google-Extended, PerplexityBot, CCBot and more), plus sitemap declaration, /llms.txt presence and Content-Signal lines. It makes at most two requests per domain (robots.txt, then /llms.txt), one after the other.

Input

FieldTypeDescription
domains (required)string[]Domains or URLs, 1–1000. Each is reduced to its origin; bare domains use https. Duplicate origins are skipped.
botsstring[]Limit the audit to these tokens. Empty (default) checks all 21.
checkPathsstring[]Up to 20 extra paths to evaluate besides / (matched against the rules, not fetched).
includeRawRobotsbooleanAdd the fetched robots.txt text (up to 500 KiB) to each item. Default false.
maxConcurrencyintegerDomains audited in parallel, 1–20. Default 5.

Output (one dataset item per domain)

The dataset's Overview view shows one row per domain with the robots.txt status, the AI-crawler summary at /, sitemap and llms.txt status. Pricing is pay-per-event: $0.01 per domain audited, charged once per successfully audited domain, plus a tiny actor-start charge ($0.00005 per GB of run memory) per run. See the actor page on Apify for current pricing.

llms.txt Validator & Generator (Apify actor)

Status: live on Apify Store. Run it on Apify (aicrawlready/llms-txt-validator-generator). Pricing: $0.004 per site validated and $0.02 per llms.txt draft generated, pay per event.

Checks whether a site publishes a valid /llms.txt against the llmstxt.org format: a 0–100 score and grade, pass/warn/fail per check, a broken-link check of the listed URLs and fix suggestions. It can also generate a draft llms.txt from the homepage and a capped sample of sitemap pages. It makes plain HTTP requests only, respects robots.txt for page sampling and link checks (token LlmsTxtValidator), and sends one request per host at a time.

Input

FieldTypeDescription
startUrls (required)URL[]Up to 500 sites per run. Each is reduced to its origin; enter a full llms.txt URL to validate a file in a subpath.
modestringvalidate, generate or both (default).
checkLinks / maxLinksToCheckboolean / integerCheck listed links (default true), up to 0–200 links (default 20).
checkLlmsFullTxtbooleanAlso look at /llms-full.txt (informational; default true).
maxSitemapPagesintegerPages sampled for the draft, 1–100 (default 15).
requestDelayMs, maxConcurrencyintegerPer-host delay (default 1000 ms) and sites in parallel (1–10, default 3).

Output

One dataset item per site with a flat summary (state, score, grade, check counts, broken links, llms-full.txt state, draft status) and a detailed validation object (checks, suggestions, parsed sections, link-check results). Generated drafts are also saved as text files in the key-value store. A site counts as validated when its llms.txt URL returned an HTTP response, including a 404 (no llms.txt found); invalid inputs, DNS failures and timeouts aren't charged. Apify also adds a tiny actor-start charge ($0.00005 per GB of run memory) per run.

AI Crawler Checker (extension for Chrome, Edge and Firefox)

Status: coming soon for Chrome, Microsoft Edge and Firefox (not on any store yet).

Click the toolbar icon on any site to see how its robots.txt treats 21 AI crawler and fetcher tokens: Allowed, Blocked or Not mentioned, with the exact rule and line number, and a link to each vendor's documentation.

Features

Permissions

PermissionWhy
activeTabRead the current tab's URL, only when you click the icon.
storageSave license state and (Pro) history locally on your device.
Optional site access (https://*/*, http://*/*)Nothing at install. The browser asks for access to the one site you check, when you press Check; revocable in the browser's extension settings.

No background service worker, content scripts, remote code or analytics. See the privacy policy.

Public API

The checkers on this site call three JSON endpoints, and you can call them too. They're free, need no key, and run on Cloudflare Workers. Please keep volumes modest: the limits below are enforced, and results can change as sites change. Base URL: https://aicrawlready.com.

Limits that apply to all three

POST /api/crawl-budget

Fetches a site's public robots.txt and XML sitemaps (up to 5 sitemaps analyzed, 8 s timeout per request) and returns a crawl-waste score with top fixes. Body: { "url": "example.com" } (scheme optional, https assumed). Responses are cached in memory for up to 5 minutes.

curl -s -X POST https://aicrawlready.com/api/crawl-budget \
  -H 'content-type: application/json' \
  -d '{"url":"example.com"}'

Response (abridged):

{
  "inputUrl": "example.com",
  "normalizedOrigin": "https://example.com",
  "fetchedAt": "2026-09-25T00:00:00.000Z",
  "robots": { "rawLength": 0, "groups": [...], "disallowHighlights": [...], "sitemapUrls": [...] } | null,
  "robotsError": "HTTP 404 fetching robots.txt",        // when robots.txt failed
  "sitemaps": { "discovered": [...], "analyzed": [{ "url": "...", "ok": true, "parsed": {...} }], "totalUrlCount": 0 },
  "freshness": { "withLastmod": 0, "withoutLastmod": 0, "staleShare": 0, "note": "..." },
  "score": { "value": 20, "label": "Low likely crawl-waste" },   // 0 = clean, 100 = high likely waste
  "topFixes": [{ "title": "...", "detail": "..." }],
  "rateLimitNote": "...",
  "cached": false
}

GET|POST /api/ai-crawler-check

Fetches only /robots.txt for the given host (8 s timeout, 500 KiB cap, up to 5 redirects, each re-checked) and reports, for each of 21 AI crawler tokens, whether / is allowed or blocked and which rule decided it. Parameter: url (query string for GET, JSON body for POST). Cached for up to 5 minutes.

curl -s 'https://aicrawlready.com/api/ai-crawler-check?url=nytimes.com'

curl -s -X POST https://aicrawlready.com/api/ai-crawler-check \
  -H 'content-type: application/json' -d '{"url":"nytimes.com"}'
{
  "input": "nytimes.com",
  "robotsUrl": "https://nytimes.com/robots.txt",
  "finalUrl": "https://www.nytimes.com/robots.txt",
  "httpStatus": 200,
  "robotsState": "ok",            // ok | not-found | unreachable
  "checkedPath": "/",
  "bytes": 0, "truncated": false, "groupCount": 0,
  "warnings": ["robots.txt redirected 1 time(s) to ..."],
  "bots": [{
    "token": "GPTBot", "vendor": "OpenAI", "purpose": "...", "docs": "https://...",
    "userTriggered": false,
    "verdict": "blocked",          // allowed | blocked
    "mention": "specific",         // specific | wildcard | none | n/a
    "rule": { "type": "disallow", "pattern": "/", "line": 0 },
    "groupAgents": ["GPTBot"], "groupLines": [0]
  }],
  "fetchedAt": "...",
  "cached": false
}

GET|POST /api/redirect-check

Follows redirects manually for up to 20 URLs per call, up to 10 redirects per URL. Each hop is a HEAD request, with a GET fallback when HEAD errors or returns 4xx/5xx (headers only; bodies aren't read), 5 s per request. Every hop's host goes through the SSRF check. Per call: at most 45 outbound requests including DNS lookups (Workers Free allows 50), 3 URLs at a time, and a 22 s time budget; URLs that hit the cap come back with a budget issue so you can re-check them. For bigger lists, send batches of about 5 URLs (the web page does this). Rate limit: 30 calls per minute per IP (in memory, best-effort). Responses aren't cached.

Parameters: GET ?url=a&url=b (repeat url), or POST { "urls": "one\nper\nline" } (a string with one URL per line, or an array of strings). Lines without a scheme are checked as https.

curl -s 'https://aicrawlready.com/api/redirect-check?url=http://github.com&url=https://example.com/'

curl -s -X POST https://aicrawlready.com/api/redirect-check \
  -H 'content-type: application/json' \
  -d '{"urls":"http://github.com\nhttps://example.com/old-page"}'
{
  "checkedAt": "...", "maxUrls": 20, "truncated": false,
  "rejected": [{ "input": "ftp://x", "reason": "Only http:// and https:// URLs are supported." }],
  "requests": 4,
  "results": [{
    "input": "http://github.com",
    "startUrl": "http://github.com/",
    "hops": [
      { "url": "http://github.com/", "status": 301, "location": "https://github.com/",
        "next": "https://github.com/", "method": "HEAD", "ms": 17, "kinds": ["http-to-https", "permanent"] },
      { "url": "https://github.com/", "status": 200, "location": null, "next": null,
        "method": "HEAD", "ms": 15, "kinds": [] }
    ],
    "redirects": 1,
    "finalUrl": "https://github.com/",
    "finalStatus": 200,
    "issues": [],   // codes: loop, too-many-hops, chain, final-4xx, final-5xx, https-downgrade,
                    // temporary-redirect, missing-location, blocked, error, budget
    "suggestedTarget": "https://github.com/"
  }]
}

Try them in the browser: Crawl Budget Snapshot, AI crawler checker, redirect chain checker. Background reading in the guides. Google's robots.txt spec and RFC 9309 describe the matching rules the checkers follow.