# robots.txt for gruxai.com
#
# WHY THIS FILE EXISTS.
# Until it did, Cloudflare Pages had no /robots.txt asset to serve, so the origin
# answered the request with index.html and Cloudflare prepended its managed block
# to 20KB of HTML. Measured 2026-08-13: 21,669 bytes, content type text/plain, an
# entire HTML document starting at line 62, and zero Sitemap directives. A real file
# fixes both halves.
#
# KEEP THIS FILE FREE OF HTML TOKENS, INCLUDING IN COMMENTS. The documented check for
# the regression above greps the served file for a doctype declaration and expects
# zero hits. Quoting that token here, even to explain the bug, makes the check report
# a failure forever and teaches the next reader to ignore it. Describe the markup,
# never spell it. This paragraph exists because the first draft of this comment did
# exactly that and turned its own verification red.
#
# CLOUDFLARE NO LONGER PREPENDS A MANAGED BLOCK, AND THAT IS A SETTING, NOT A DEFAULT.
# Until 2026-09-13 this file was served with Cloudflare's managed robots.txt block on
# top of it, carrying a `Content-Signal` line and nine `Disallow: /` groups. The note
# that used to sit here said the merge was a no-op because both halves said the same
# thing. That was true only while this file also said no, and it is the reason the
# managed block went unexamined for a month: it agreed with us, so nobody read it.
#
# Managed robots.txt is now OFF for the zone, under AI Crawl Control, Signals. This
# file is the whole of what is served, and `curl -sS https://gruxai.com/robots.txt`
# returning anything that is not byte identical to it means somebody turned it back on.
#
# THE HOUSE RULE THIS FILE ENCODES.
#   1. Never Disallow a path that also serves noindex. The two directives do
#      different jobs: noindex means do not LIST this and requires the crawler to
#      FETCH the page to read it, while Disallow means do not FETCH. Both on one path
#      cancels the first. This site serves noindex on nothing (measured: 0 hits for
#      noindex in index.html), so the conflict cannot arise here. It arises the moment
#      someone adds a private page, so read this before adding a Disallow.
#   2. Never Disallow a URL cited as evidence. Anything named as proof in a filing,
#      a registration, a store listing or a support ticket must be fetchable by `*`.
#   3. Every named group repeats its FULL rule set. A crawler obeys only its single
#      most specific matching group and inherits NOTHING from `*`, so a named group
#      containing only `Allow: /` silently exempts that bot from every rule in the
#      file. That is the bug that shipped on Thudletter (14 groups) and AI Anyone.
#      There are now no named groups at all, which satisfies the rule by having
#      nothing to get wrong. Add one and you own repeating every rule into it.
#
# THE DECISION, REVERSED DELIBERATELY ON 2026-09-13, BY JACK.
# This file used to Disallow nine AI crawlers by name: Amazonbot, anthropic-ai,
# Applebot-Extended, Bytespider, CCBot, ClaudeBot, Google-Extended, GPTBot and
# meta-externalagent. The comment that stood here argued both sides and closed with
# "withholding a corpus is reversible in one commit while a training run is not."
# This is that commit. The old reasoning is preserved below rather than deleted,
# because a reversal with no record of what it reversed teaches the next reader
# nothing and invites a third swing.
#
# THE ARGUMENT THAT WON, and it was already written down as the counter-case:
# Grux OS is a free, MIT licensed tool FOR people who work with AI agents, and the
# question it answers best is one that gets asked INSIDE an assistant. "How do I run
# an AI agent locally on a Mac", "how do I stop my Swift agent leaking secrets",
# "which local model fits 48 GB". Being retrievable when somebody asks that is the
# distribution channel, not a leak.
#
# THE ARGUMENT THAT LOST, and why it was weaker than it looked:
# There is no corpus here to withhold. The entire source tree is public on GitHub
# under an MIT licence, and has been since 2026-08-22. Blocking a training crawler
# from a marketing site whose product is an open repository withholds the brochure
# and hands over the building. It bought no protection and cost the whole channel.
#
# WHAT WAS MEASURED, 2026-09-13, and what made this urgent rather than tidy:
#   - Google Search Console, Generative AI features, 180 days: ONE impression.
#   - Cloudflare firewall events: `block` from the managed AI rule against
#     OAI-SearchBot, ChatGPT-User, PerplexityBot, Perplexity-User, DuckAssistBot,
#     YouBot, ClaudeBot, Claude-User and GPTBot. Six of those were never named in
#     this file and are inference-time or user-triggered fetchers, which the comment
#     above explicitly believed were "NOT blocked". They were, at the edge, by a
#     zone setting this file could not see and did not mention.
#   - AI Labyrinth was ON, actively serving decoy pages to AI crawlers.
#   - The deprecated "Block AI bots" rule was set to extend to MIXED PURPOSE
#     crawlers on 2026-09-15, which would have taken search indexing with it.
# All four are now off. robots.txt was the smallest of the four locks and the only
# one anybody would have found by reading this repository.
#
# THERE ARE NO NAMED GROUPS NOW. Everything matches `*` and everything is allowed.
# If a crawler ever has to be blocked again, block it HERE and at the edge in the
# same change, and say which of the two is load bearing. One without the other is
# how this site spent a month believing it was open while returning 403.

User-agent: *
Allow: /

# A Sitemap record is not scoped to a user-agent group, so it applies to every
# crawler and survives Cloudflare prepending its managed block above this line.
Sitemap: https://gruxai.com/sitemap.xml