Crawlable Is Not Citable: A Practical GEO Baseline for Technical Sites

A technical site can be perfectly crawlable and still be unhelpful as a source. A bot can reach the page, parse the heading, and index the URL, yet have no reason to surface it when someone asks a precise question.

That is the useful way to think about generative-search work. The baseline is not a bag of AI-crawler directives. It is an evidence chain: the page is reachable, the canonical version is clear, the claim is specific, the author and date are visible, and the source material is close enough for a reader to inspect.

Crawl control has a narrow job

A robots.txt file tells crawlers which URLs they may fetch. It is primarily a crawl-traffic control, not a way to make a sensitive web page disappear from search. If a page must not be visible, use authentication or an index-control mechanism appropriate to the page instead.

For a public technical site, the usual baseline is simple:

• allow the public pages that should be discoverable;
• disallow private, administrative, and internal API paths;
• expose a sitemap at a stable URL; and
• avoid blocking JavaScript, CSS, or image resources needed to understand a public page.

The important operational check is not only reading the source file. Request the deployed /robots.txt, /sitemap.xml, and representative public pages as a normal browser and as the crawler user agents you support. A production proxy, cache, or WAF can behave differently from local code.

Make one canonical answer per question

Generative systems and human readers both struggle when a site has five near-identical pages with small wording changes. Choose one canonical URL for the primary answer, set the canonical relation, and use internal links to make its purpose unambiguous.

For multilingual sites, language variants should be genuine translations or adaptations, not duplicated English pages with a locale in the path. Hreflang links help search engines understand the relationship, but they do not rescue thin or mismatched versions.

Evidence is the citation layer

The strongest article structure for a technical claim is predictable:

1. State the claim narrowly enough to test.
2. Explain the system boundary and the trade-off.
3. Link to the primary documentation, standard, benchmark, or incident record.
4. Say what was verified in production and what remains an assumption.
5. Update the date when the facts change.

That structure is more useful than keyword repetition. It gives a reader a path to verify the work and gives an AI system a coherent piece of context rather than scattered slogans.

Use metadata to remove ambiguity

Metadata is not a substitute for content, but it prevents needless ambiguity. Every publishable article should have a descriptive title and description, a canonical URL, an Open Graph image, visible author and publication date, and valid BlogPosting structured data. A sitemap makes the inventory discoverable; an RSS feed gives subscribers and aggregators a predictable update stream.

An llms.txt file can be a useful human-readable map of a site’s main services and strongest articles. It is not an official indexing protocol and should never be the only path to the content. Keep the actual pages, internal links, sitemap, and metadata correct first.

A release checklist that scales

Before publishing an article or changing a technical content template:

• Fetch the public URL and confirm a 200 response, correct canonical, title, description, and Open Graph image.
• Confirm the article is in the sitemap and the feed when it is published.
• Validate the JSON-LD against the visible page, not against an unpublished draft.
• Test a missing slug returns a real non-indexable response.
• Keep admin and API routes unavailable to crawlers and unauthenticated users.
• Record the source links and the date you checked them.

This is deliberately unglamorous. It is also the work that lets genuinely useful pages survive framework changes, cache changes, new search surfaces, and the next person who needs to audit the site.

Sources: Google Search Central: Introduction to robots.txt, Google: Overview of crawlers and fetchers, Google Search Central: Preferred Sources.