# FILEDAR — https://filedar.ai # Generated at build time from src/lib/seo.ts. Do not edit this file: # it is overwritten by every `npm run build`. # The index is open. # # /congress/* REAL. Real members of Congress, real STOCK Act filings, # as filed. These records carry the 5 U.S.C. §13107 # (formerly Ethics in Government Act §105) restriction on use. # # / The home: the newest disclosure in the record as its # receipt, the seven newest, a lanes line (Congress live, # SEC filings not live) and the funding statement. # # the SEC NOT LIVE. Named on /almanac, the SEC state page; no # lanes page publishes a result, a rate or a filing from them. # # No page on this site carries sample data. # ── AI CRAWLER POLICY: three classes, three answers ─────────────────── # # 1. SEARCH INDEXING — ALLOW. # Googlebot, Applebot, Bingbot # They crawl, they link, a reader arrives. This is the acquisition # engine and the reason the congressional lane is free. # # 2. AI RETRIEVAL / ANSWER — ALLOW. # OAI-SearchBot, ChatGPT-User, Claude-SearchBot, Claude-User, PerplexityBot, Perplexity-User, meta-webindexer, meta-externalfetcher # They fetch to answer a live query, and they cite and link back. # Being the source an AI points at is distribution we cannot otherwise # buy, and it is what this product IS: the citable record behind a # number. Refusing them would be refusing the shelf we are built for. # # 3. AI TRAINING — DISALLOW. # GPTBot, ClaudeBot, CCBot, meta-externalagent, Applebot-Extended # No attribution, no link, no reader. Pure extraction. # # 4. Google-Extended — ALLOW, against the pattern above, on purpose. # It BUNDLES training with grounding in Gemini Apps and Vertex AI — # a surface that cites. Blocking it to deny the training also denies # the citation, and the citation is what we are for. The training is # rent on an answer surface. # # ⚠️ It does NOT govern AI Overviews. Google states that AI is built # into Search and that robots.txt for GOOGLEBOT is the control for # Search AI features; AI Overviews and AI Mode have no separate # opt-out and are bought by allowing Googlebot, which we do. If you # are re-deriving this policy, that correction is the part that is # easy to get wrong in the confident direction. # # Do not collapse these classes. A wrong or missing token is a SILENT # no-op: the rule never matches and the file reads as a policy while # enforcing nothing. Tokens are verified in src/lib/seo.ts CRAWLERS. # ── Default. Everything not named below. ────────────────────────────── # The excluded routes are the registry's exceptions on this face — from # PAGE_STATUS in src/lib/seo.ts, read by pageStatus(): the error page, the # confirmation landings, the unopened design pages, the previewed page. # Not destinations, so not crawl budget. User-agent: * Disallow: /404 Disallow: /ballot Disallow: /ballot/census Disallow: /ballot/grammar Disallow: /ballot/retrofit Disallow: /confirm/already-on-file Disallow: /confirm/confirmed Disallow: /confirm/error Disallow: /confirm/expired Disallow: /confirm/invalid Disallow: /pricing Allow: / # ── ALLOWED, and named rather than left to the group above. ─────────── # These groups are redundant while `*` allows, DELIBERATELY: a named group # replaces `*` for that crawler, so writing them out means the answer for # each class is stated in the file rather than inferred from a default — # and it survives a future edit that tightens `*`. Each repeats the stub # exclusions because a named group inherits nothing. User-agent: Googlebot Disallow: /404 Disallow: /ballot Disallow: /ballot/census Disallow: /ballot/grammar Disallow: /ballot/retrofit Disallow: /confirm/already-on-file Disallow: /confirm/confirmed Disallow: /confirm/error Disallow: /confirm/expired Disallow: /confirm/invalid Disallow: /pricing Allow: / # search: Google Search, Images, News, Discover — and the AI Overviews and AI Mode surfaces, which have no separate opt-out User-agent: Applebot Disallow: /404 Disallow: /ballot Disallow: /ballot/census Disallow: /ballot/grammar Disallow: /ballot/retrofit Disallow: /confirm/already-on-file Disallow: /confirm/confirmed Disallow: /confirm/error Disallow: /confirm/expired Disallow: /confirm/invalid Disallow: /pricing Allow: / # search: Apple search results (Siri, Spotlight) User-agent: Bingbot Disallow: /404 Disallow: /ballot Disallow: /ballot/census Disallow: /ballot/grammar Disallow: /ballot/retrofit Disallow: /confirm/already-on-file Disallow: /confirm/confirmed Disallow: /confirm/error Disallow: /confirm/expired Disallow: /confirm/invalid Disallow: /pricing Allow: / # search: Bing search. Listed because the policy is about it; if the token is wrong the rule is a no-op and Bing falls through to the `*` group, which allows — so the failure here is benign, which is exactly why it is tolerable in the ALLOW class and would not be in the DISALLOW class User-agent: OAI-SearchBot Disallow: /404 Disallow: /ballot Disallow: /ballot/census Disallow: /ballot/grammar Disallow: /ballot/retrofit Disallow: /confirm/already-on-file Disallow: /confirm/confirmed Disallow: /confirm/error Disallow: /confirm/expired Disallow: /confirm/invalid Disallow: /pricing Allow: / # retrieval: surfaces websites in ChatGPT search results User-agent: ChatGPT-User Disallow: /404 Disallow: /ballot Disallow: /ballot/census Disallow: /ballot/grammar Disallow: /ballot/retrofit Disallow: /confirm/already-on-file Disallow: /confirm/confirmed Disallow: /confirm/error Disallow: /confirm/expired Disallow: /confirm/invalid Disallow: /pricing Allow: / # retrieval: fetches a page because a user asked; not automatic crawling User-agent: Claude-SearchBot Disallow: /404 Disallow: /ballot Disallow: /ballot/census Disallow: /ballot/grammar Disallow: /ballot/retrofit Disallow: /confirm/already-on-file Disallow: /confirm/confirmed Disallow: /confirm/error Disallow: /confirm/expired Disallow: /confirm/invalid Disallow: /pricing Allow: / # retrieval: improves search result quality for Claude users User-agent: Claude-User Disallow: /404 Disallow: /ballot Disallow: /ballot/census Disallow: /ballot/grammar Disallow: /ballot/retrofit Disallow: /confirm/already-on-file Disallow: /confirm/confirmed Disallow: /confirm/error Disallow: /confirm/expired Disallow: /confirm/invalid Disallow: /pricing Allow: / # retrieval: fetches a page in response to a Claude user question User-agent: PerplexityBot Disallow: /404 Disallow: /ballot Disallow: /ballot/census Disallow: /ballot/grammar Disallow: /ballot/retrofit Disallow: /confirm/already-on-file Disallow: /confirm/confirmed Disallow: /confirm/error Disallow: /confirm/expired Disallow: /confirm/invalid Disallow: /pricing Allow: / # retrieval: surfaces and LINKS websites in Perplexity results; stated not to be used for training User-agent: Perplexity-User Disallow: /404 Disallow: /ballot Disallow: /ballot/census Disallow: /ballot/grammar Disallow: /ballot/retrofit Disallow: /confirm/already-on-file Disallow: /confirm/confirmed Disallow: /confirm/error Disallow: /confirm/expired Disallow: /confirm/invalid Disallow: /pricing Allow: / # retrieval: user-initiated retrieval. ⚠️ Perplexity has stated this one is "an agent, not a bot" and is not bound by robots.txt, and Cloudflare reported undeclared crawlers evading directives in Aug 2025 — so this row is a STATEMENT OF OUR POLICY, not a belief about their compliance User-agent: meta-webindexer Disallow: /404 Disallow: /ballot Disallow: /ballot/census Disallow: /ballot/grammar Disallow: /ballot/retrofit Disallow: /confirm/already-on-file Disallow: /confirm/confirmed Disallow: /confirm/error Disallow: /confirm/expired Disallow: /confirm/invalid Disallow: /pricing Allow: / # retrieval: improves Meta AI search results User-agent: meta-externalfetcher Disallow: /404 Disallow: /ballot Disallow: /ballot/census Disallow: /ballot/grammar Disallow: /ballot/retrofit Disallow: /confirm/already-on-file Disallow: /confirm/confirmed Disallow: /confirm/error Disallow: /confirm/expired Disallow: /confirm/invalid Disallow: /pricing Allow: / # retrieval: agentic / user-requested link evaluation User-agent: Google-Extended Disallow: /404 Disallow: /ballot Disallow: /ballot/census Disallow: /ballot/grammar Disallow: /ballot/retrofit Disallow: /confirm/already-on-file Disallow: /confirm/confirmed Disallow: /confirm/error Disallow: /confirm/expired Disallow: /confirm/invalid Disallow: /pricing Allow: / # mixed: BUNDLES training with grounding. Governs Gemini model training AND grounding in Gemini Apps and Vertex AI — a surface that cites. Allowed on purpose; see the note below # ── DISALLOWED — AI training. ───────────────────────────────────────── # No attribution, no link, no reader. Pure extraction of a corpus whose # value is that it is ours and sourced. Every token here was verified # against the operator's own documentation: in this class a wrong token is # a silent no-op that leaves the crawler with the whole site. User-agent: GPTBot Disallow: / # crawls content that may be used to train foundation models User-agent: ClaudeBot Disallow: / # collects web content for model development User-agent: CCBot Disallow: / # builds the Common Crawl corpus, which is a standard training input User-agent: meta-externalagent Disallow: / # crawls for training foundation AI models User-agent: Applebot-Extended Disallow: / # directive-only token; does not crawl, it governs whether Applebot's crawl may train Apple foundation models. Disallowing it does not remove us from Apple search Sitemap: https://filedar.ai/sitemap.xml