vikarti.anatra/main
1
0
Fork 0
mirror of https://github.com/manualdousuario/marreta.git synced 2025-09-02 02:30:20 +00:00
Code Issues Projects Releases Packages Wiki Activity Actions

adicionado robots bloqueando indexação de paginas internas e ias

Browse source
This commit is contained in:
Renan Bernordi 2024-11-28 17:02:56 -03:00
parent 008b11f42f
commit a4cfef2d40
1 changed files with 34 additions and 0 deletions
Show stats Download patch file Download diff file Expand all files Collapse all files

34
app/robots.txt Normal file
View file

@ -0,0 +1,34 @@
User-agent: *
Disallow: /p/
Disallow: /api/
# Block common AI crawlers
User-agent: GPTBot
Disallow: /
User-agent: ChatGPT-User
Disallow: /
User-agent: anthropic-ai
Disallow: /
User-agent: CCBot
Disallow: /
User-agent: Google-Extended
Disallow: /
User-agent: FacebookBot
Disallow: /
User-agent: Claude-Web
Disallow: /
User-agent: Omgilibot
Disallow: /
User-agent: Omgili
Disallow: /
User-agent: FacebookBot
Disallow: /