xiaoxue_admin
/
crawlee-python
1
0
Fork 0
Code Issues Pull Requests Packages Projects Releases Wiki Activity
crawlee-python/website/static/robots.txt

3 lines
62 B
Plaintext
Raw Permalink Blame History

User-agent: *
Sitemap: https://crawlee.dev/python/sitemap.xml
Reference in New Issue View Git Blame Copy Permalink