Files
supervision/docs/robots.txt
T
c4a3038c57 feat(docs): add GEO infrastructure for AI search visibility (#2224)
* feat: add GEO infrastructure for AI search visibility
* fix: guard GEO blocks with {% if page %} to fix 404 build error
* feat: GEO platform optimization — FAQPage, IndexNow, answer paragraphs
* fix: move Disallow:/0.*/ to User-agent:* in robots.txt
* fix: block all numeric versioned paths in robots.txt

---------

Co-authored-by: Claude Code <noreply@anthropic.com>
Co-authored-by: pre-commit-ci[bot] <66853113+pre-commit-ci[bot]@users.noreply.github.com>
2026-04-21 11:40:44 +02:00

35 lines
608 B
Plaintext

User-agent: *
Allow: /
# Block all numeric versioned paths (/0.28.0/, /1.0.0/, etc.) — canonical URL is /latest/
# robots.txt has no character classes, so one Disallow per leading digit is required.
Disallow: /0.*/
Disallow: /1.*/
Disallow: /2.*/
Disallow: /3.*/
Disallow: /4.*/
Disallow: /5.*/
Disallow: /6.*/
Disallow: /7.*/
Disallow: /8.*/
Disallow: /9.*/
User-agent: GPTBot
Allow: /
User-agent: ClaudeBot
Allow: /
User-agent: PerplexityBot
Allow: /
User-agent: Bytespider
Allow: /
User-agent: CCBot
Allow: /
User-agent: GoogleOther
Allow: /
Sitemap: https://supervision.roboflow.com/sitemap.xml