# robots.txt for ioai.tech # DRAFT — one decision to confirm before serving. See "THE DECISION" below. # Last reviewed: 2026-09-22 # # --------------------------------------------------------------------------- # THE DECISION (IOAI leadership, not engineering, not the agency) # # Allowing AI crawlers is a content-licensing choice, not a technical one. # A company with a large proprietary archive gives something up by allowing # them. IOAI does not have one: this site is marketing copy plus machine- # readable descriptors that exist precisely to be read by machines. # # Recommendation: ALLOW EVERYTHING, explicitly and by name. Being found by # agents is the strategy, not a side effect of it. # # Revisit if IOAI later publishes technical documentation, research or a # corpus with independent licensing value. Until then, every Disallow costs # reach and buys nothing. # --------------------------------------------------------------------------- # # NOTE: robots.txt is a request, not access control. It does not protect # anything. Never put a path here that you need kept private — listing it # advertises it. Use authentication for that. # --- Default: everyone, everything --------------------------------------- User-agent: * Allow: / # --- AI and agent crawlers, named explicitly ------------------------------ # # Several of these treat an unstated rule as a reason to skip rather than a # reason to proceed. Silence is not permission. Naming them costs nothing and # removes the ambiguity. User-agent: GPTBot Allow: / User-agent: OAI-SearchBot Allow: / User-agent: ChatGPT-User Allow: / User-agent: ClaudeBot Allow: / User-agent: Claude-User Allow: / User-agent: Claude-SearchBot Allow: / User-agent: Google-Extended Allow: / User-agent: PerplexityBot Allow: / User-agent: Perplexity-User Allow: / User-agent: Applebot Allow: / User-agent: Applebot-Extended Allow: / User-agent: meta-externalagent Allow: / User-agent: Amazonbot Allow: / User-agent: Bytespider Allow: / User-agent: CCBot Allow: / User-agent: cohere-ai Allow: / User-agent: Diffbot Allow: / User-agent: YouBot Allow: / User-agent: Timpibot Allow: / User-agent: omgili Allow: / # --- Paths that must never be disallowed ---------------------------------- # # These are listed as a standing warning to whoever edits this file next. # Blocking any of them silently breaks agent discovery, and the failure is # invisible until somebody wonders why traffic never arrived. # # /llms.txt # /.well-known/ (agent-card.json, mcp.json) # /openapi.json # /sitemap.xml # # A blanket "Disallow: /.well-known/" is the most common version of this # mistake. Do not add one. # --- Housekeeping --------------------------------------------------------- # # Crawl-delay is ignored by Google and most major crawlers. If a crawler is # actually costing money, rate-limit it at the edge; a line here will not. # [CONFIRM] Add Disallow rules here ONLY for genuinely worthless paths — # search result pages, session URLs, staging mirrors. Nothing in the current # site map qualifies. # --- Sitemap -------------------------------------------------------------- # # Publishing this line is not the same as the sitemap being fetched. Register # it in the search consoles and push changes via IndexNow, then check the # fetch logs. An unfetched sitemap is the most common silent failure in the # whole discovery stack. Sitemap: https://ioai.tech/sitemap.xml