* feat: add GEO infrastructure for AI search visibility
* fix: guard GEO blocks with {% if page %} to fix 404 build error
* feat: GEO platform optimization — FAQPage, IndexNow, answer paragraphs
* fix: move Disallow:/0.*/ to User-agent:* in robots.txt
* fix: block all numeric versioned paths in robots.txt
---------
Co-authored-by: Claude Code <noreply@anthropic.com>
Co-authored-by: pre-commit-ci[bot] <66853113+pre-commit-ci[bot]@users.noreply.github.com>
35 lines
608 B
Plaintext
35 lines
608 B
Plaintext
User-agent: *
|
|
Allow: /
|
|
# Block all numeric versioned paths (/0.28.0/, /1.0.0/, etc.) — canonical URL is /latest/
|
|
# robots.txt has no character classes, so one Disallow per leading digit is required.
|
|
Disallow: /0.*/
|
|
Disallow: /1.*/
|
|
Disallow: /2.*/
|
|
Disallow: /3.*/
|
|
Disallow: /4.*/
|
|
Disallow: /5.*/
|
|
Disallow: /6.*/
|
|
Disallow: /7.*/
|
|
Disallow: /8.*/
|
|
Disallow: /9.*/
|
|
|
|
User-agent: GPTBot
|
|
Allow: /
|
|
|
|
User-agent: ClaudeBot
|
|
Allow: /
|
|
|
|
User-agent: PerplexityBot
|
|
Allow: /
|
|
|
|
User-agent: Bytespider
|
|
Allow: /
|
|
|
|
User-agent: CCBot
|
|
Allow: /
|
|
|
|
User-agent: GoogleOther
|
|
Allow: /
|
|
|
|
Sitemap: https://supervision.roboflow.com/sitemap.xml
|