mirror of
https://github.com/mitchellkrogza/nginx-ultimate-bad-bot-blocker.git
synced 2025-09-02 02:29:58 +00:00
142 lines
6.4 KiB
Markdown
142 lines
6.4 KiB
Markdown
# Robots.txt Directives
|
|
|
|
The robots.txt file residing in this directory should be used in addition to using the Nginx Bad Bot Blocker.
|
|
|
|
There is good reason for this.
|
|
|
|
It is considered good practice to actually tell a bot whether it is allowed or denied access.
|
|
|
|
So as a webmaster or server operator it is the right and also responsible thing to tell any of our listed bad bots to go away first.
|
|
|
|
If a bot ignores robots.txt (which many do) it will then be blocked outright by the blocker, make no mistake about that.
|
|
|
|
************************************************
|
|
## Help Support This Project
|
|
|
|
[<img src="https://github.com/mitchellkrogza/nginx-ultimate-bad-bot-blocker/blob/master/.assets/kofi4.png" alt="Buy me Coffee" width="300"/>](https://ko-fi.com/mitchellkrog)
|
|
|
|
************************************************
|
|
- Created by: https://github.com/mitchellkrogza for use on Nginx Web Server https://www.nginx.com/
|
|
- Copyright Mitchell Krog <mitchellkrog@gmail.com>
|
|
|
|
Not Using Nginx? See the [](https://github.com/mitchellkrogza/apache-ultimate-bad-bot-blocker)
|
|
|
|
************************************************
|
|
## Optimized for Wordpress Sites
|
|
|
|
This robots.txt file is designed primarily for wordpress sites.
|
|
This is revealed in the starting lines.
|
|
|
|
```
|
|
User-agent: *
|
|
Disallow: /wp-admin/
|
|
Allow: /wp-admin/admin-ajax.php
|
|
```
|
|
|
|
This means all user agents are ALLOWED to access all parts of the web sites by default
|
|
- they are NOT allowed to crawl the Wordpress /wp-admin/ folder
|
|
- they ARE allowed to access /wp-admin/admin-ajax.php
|
|
|
|
You can modify these beginning lines to your liking.
|
|
|
|
Then after the beginning lines we start explicitly telling bots to go away.
|
|
|
|
Example:
|
|
|
|
```
|
|
User-agent: 360Spider
|
|
Disallow:/
|
|
User-agent: 404checker
|
|
Disallow:/
|
|
.....
|
|
.....
|
|
.....
|
|
.....
|
|
```
|
|
|
|
************************************************
|
|
##### Keeping your robots.txt updated
|
|
|
|
Check out the updaterobots.sh bash script for a simple updater for all your web sites.
|
|
|
|
|
|
************************************************
|
|
##### Follow me on twitter @ubuntu101za
|
|
|
|
<a href='https://twitter.com/ubuntu101za'><img src='https://img.shields.io/twitter/follow/ubuntu101za.svg?style=social&label=Follow' alt='Follow @ubuntu101za'></a>
|
|
|
|
************************************************
|
|
### SOME OTHER AWESOME FREE PROJECTS
|
|
|
|
- https://github.com/mitchellkrogza/apache-ultimate-bad-bot-blocker
|
|
- https://github.com/mitchellkrogza/Badd-Boyz-Hosts
|
|
- https://github.com/mitchellkrogza/Stop.Google.Analytics.Ghost.Spam.HOWTO
|
|
- https://github.com/mitchellkrogza/The-Big-List-of-Hacked-Malware-Web-Sites
|
|
- https://github.com/mitchellkrogza/fail2ban-useful-scripts
|
|
- https://github.com/mitchellkrogza/linux-server-administration-scripts
|
|
- https://github.com/mitchellkrogza/Travis-CI-Nginx-for-Testing-Nginx-Configuration
|
|
- https://github.com/mitchellkrogza/Travis-CI-for-Apache-For-Testing-Apache-and-PHP-Configurations
|
|
- https://github.com/mitchellkrogza/Fail2Ban-Blacklist-JAIL-for-Repeat-Offenders-with-Perma-Extended-Banning
|
|
- https://github.com/funilrys/funceble
|
|
|
|
************************************************
|
|
### INTO PHOTOGRAPHY?
|
|
|
|
Come drop by and visit me at https://mitchellkrog.com or <a href='https://twitter.com/MitchellKrog'><img src='https://img.shields.io/twitter/follow/MitchellKrog.svg?style=social&label=Follow' alt='Follow @MitchellKrog'></a>
|
|
|
|
************************************************
|
|
### ACKNOWLEDGMENTS & CONTRIBUTORS:
|
|
|
|
Many Thanks to those contributing to this project.
|
|
|
|
Many parts of the generator scripts and code running behind this project have been adapted from snippets from hundreds of sources.
|
|
In fact it is so hard to mention everyone but here are a few key people whose little snippets of code have helped me introduce new features all the time. Show them some love and check out some of their projects too.
|
|
|
|
- Stuart Cardall - https://github.com/itoffshore (Install, Update and Setup Scripts & Alpine Linux Package Maintainer)
|
|
- Nissar Chababy - https://github.com/funilrys/funceble (Excellent script for checking ACTIVE, INACTIVE and EXPIRED Domain Names)
|
|
- Marius Voila https://github.com/mariusv
|
|
- Cătălin Mariș https://github.com/alrra
|
|
- deformhead https://github.com/deformhead
|
|
- bluedragonz https://github.com/bluedragonz
|
|
- Alexander https://github.com/shoonois
|
|
- Steven Black https://github.com/StevenBlack
|
|
- Fail2Ban - https://github.com/fail2ban
|
|
- Stevie-Ray Hartog https://github.com/Stevie-Ray
|
|
- Sir Athos from StackOverFlow - http://stackoverflow.com/users/2245910/sir-athos (help with Travis Build Tagging and Committing)
|
|
- StackOverflow - http://stackoverflow.com/ (bash scripts from hundreds of questions and answers)
|
|
- SuperUser - http://superuser.com/ (snippets from various questions and answers)
|
|
|
|
If you believe your name should be here, drop me a line.
|
|
|
|
************************************************
|
|
### Writing Code like this takes lots of time !!
|
|
|
|
Thousand of hours of programming and testing have gone into this project, show some love
|
|
|
|
[<img src="https://github.com/mitchellkrogza/nginx-ultimate-bad-bot-blocker/blob/master/.assets/kofi4.png" alt="Buy me Coffee" width="300"/>](https://ko-fi.com/mitchellkrog)
|
|
|
|
<img src="https://github.com/mitchellkrogza/nginx-ultimate-bad-bot-blocker/blob/master/.assets/zuko.png"/>
|
|
|
|
************************************************
|
|
# MIT License
|
|
|
|
## Copyright (c) 2017 Mitchell Krog - mitchellkrog@gmail.com
|
|
## https://github.com/mitchellkrogza
|
|
|
|
Permission is hereby granted, free of charge, to any person obtaining a copy
|
|
of this software and associated documentation files (the "Software"), to deal
|
|
in the Software without restriction, including without limitation the rights
|
|
to use, copy, modify, merge, publish, distribute, sublicense, and/or sell
|
|
copies of the Software, and to permit persons to whom the Software is
|
|
furnished to do so, subject to the following conditions:
|
|
|
|
The above copyright notice and this permission notice shall be included in all
|
|
copies or substantial portions of the Software.
|
|
|
|
THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
|
|
IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
|
|
FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE
|
|
AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER
|
|
LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM,
|
|
OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE
|
|
SOFTWARE.
|