Commit graph

119 commits

Author SHA1 Message Date
Mitchell Krog
c895de7ddd
Remove x22 cases - Require Exact Matching 2019-04-02 11:05:27 +02:00
Travis
32c69933b0 V3.2019.04.1358 [ci skip] 2019-04-01 13:31:58 +02:00
Mitchell Krog
a6e0e00f05
Update bad-user-agents.list + bad-referrers.list 2019-04-01 13:29:33 +02:00
Travis
e5db42a129 V3.2019.03.1355 [ci skip] 2019-03-28 11:53:17 +02:00
Mitchell Krog
74f4d6f9cb
Update bad-user-agents.list 2019-03-28 11:21:28 +02:00
Mitchell Krog
48e99f5b29 Add Mediatoolkitbot
Closes: #245
2019-01-25 15:10:51 +02:00
Ondrej Simek
6b9c46f47f Remove Seznam.cz from bad user agents. 2019-01-11 11:05:01 +01:00
Travis
a2aeeb4932 V3.2018.08.1176 [ci skip] 2018-08-24 16:57:07 +02:00
Mitchell Krog
14dbf10a21
User-Agents (+)
Closes: #206
2018-08-24 16:54:04 +02:00
Travis
bd6c13629c V3.2018.08.1155 [ci skip] 2018-08-09 11:36:04 +02:00
Mitchell Krog
b326f13416
User-Agent (+) Tracemyfile
REF: https://github.com/mitchellkrogza/apache-ultimate-bad-bot-blocker/pull/93
REF: https://www.webmasterworld.com/search_engine_spiders/4901456.htm
2018-08-09 11:33:10 +02:00
Travis
3bc0f89a34 V3.2018.08.1150 [ci skip] 2018-08-05 17:45:22 +02:00
Mitchell Krog
4582583051
Merge pull request #182 from Jos512/fyberspider
Added FyberSpider bot
2018-08-05 17:42:56 +02:00
Mitchell Krog
9f4914790f
Merge pull request #183 from Jos512/archiveBot
Add archive.org bot
2018-08-05 17:37:14 +02:00
Mitchell Krog
24545724ed
Update User-Agent OpenVAS 2018-07-24 09:08:44 +02:00
Jos512
9778d57afd Added archive bot 2018-06-16 14:27:58 +02:00
Jos512
c380089e90 Added FyberSpider bot 2018-06-16 14:23:23 +02:00
Travis
2a3bc40f73 V3.2018.06.1113 [ci skip] 2018-06-08 10:44:03 +02:00
Mitchell Krog
ec919d888f
Bot (+) Quick-Crawler
Referrer (+) scrapinghub.com
Closes: #181
2018-06-08 10:28:26 +02:00
Travis
cdc76609d9 V3.2018.06.1112 [ci skip] 2018-06-07 11:21:22 +02:00
Mitchell Krog
ad5750f32d
Bot (+) Referrer (+)
- PXBroker / db.speedup.gdn
Closes: #180
2018-06-07 11:12:56 +02:00
Travis
a9ef73b12b V3.2018.06.1109 [ci skip] 2018-06-06 16:46:52 +02:00
Mitchell Krog
fe2647c0e6
User-Agent + - Mr.4x3
Closes: #177
2018-06-06 16:36:40 +02:00
Mitchell Krog
99a41f81e3
Remove "NYU" already covered by "scan.lol"
-REF: ab1c59ad02 (commitcomment-29255444)
2018-06-06 08:40:45 +02:00
Mark Beech
f5fdf6ec02
Update bad-user-agents.list 2018-05-16 21:44:27 +01:00
Travis
fd87312e39 V3.2018.04.1098 [ci skip] 2018-04-21 13:23:48 +02:00
Mitchell Krog
23fb3f62c6
Referrers (+) / Bots (+) 2018-04-21 13:17:11 +02:00
Travis
5d04fa114d V3.2018.04.1095 [ci skip] 2018-04-20 11:23:39 +02:00
Mitchell Krog
b4795ed886
Bots (+)
OutclicksBot
2018-04-20 11:16:50 +02:00
Travis
9ead97e9d6 V3.2018.04.1094 [ci skip] 2018-04-20 09:28:10 +02:00
Mitchell Krog
4f83090d8c
Bots (+)
Zauba Search for Research; http://www.zauba.io/; admin@zauba.io
- Does not request robots.txt
2018-04-20 09:21:32 +02:00
Travis
83596094ae V3.2018.04.1091 [ci skip] 2018-04-14 16:58:58 +02:00
Mitchell Krog
5a75d55c5e
Merge pull request #153 from StarkRavingZA/master
Request Re-Addition of RankingBot
2018-04-14 16:55:41 +02:00
Travis
1611ce1591 V3.2018.04.1090 [ci skip] 2018-04-13 09:00:19 +02:00
Mitchell Krog
c56c381dd3
Bots (+)
IndeedBot

Closes: #155
2018-04-13 08:53:37 +02:00
Travis
bae8a933c6 V3.2018.04.1082 [ci skip] 2018-04-03 09:41:40 +02:00
Mitchell Krog
98623076e4
Bots (+)
crawler4j
2018-04-03 09:35:30 +02:00
StarkRavingZA
97fa50811e Request Re-Addition of RankingBot 2018-03-31 16:47:40 +02:00
Travis
8ab7b68053 V3.2018.03.1078 [ci skip] 2018-03-31 12:27:12 +02:00
Mitchell Krog
aec2eeaaa8
Referrers (+) / Bots (+)
REFERRERS
veselokloun.ru
edudocs.net
exdocsfiles.com
pillscheap24h.com
animenime.ru
sundrugstore.com
seocheckupx.com
game300.ru
hccoder.info
buntube.net
filesclub.net
saneyes.com
compiko.info
bitcoinpile.com
docs4all.com
infodocsportal.com
mixx.com
vzube.com
yun56.co

BOTS

scan.lol
NYU
MauiBot
2018-03-31 12:21:10 +02:00
Travis
25fd2ae663 V3.2018.03.1074 [ci skip] 2018-03-29 09:23:06 +02:00
Mitchell Krog
4341d84d24
Bots (+) Closes: #146
crawler - crawler.feedback@gmail.com
2018-03-29 09:17:12 +02:00
Travis
4963244255 V3.2018.03.1073 [ci skip] 2018-03-27 11:40:03 +02:00
StarkRavingZA
fcdb5bf11e Add new User Agents and Referer 2018-03-27 11:19:09 +02:00
Travis
f46621d1a2 V3.2018.03.1069 [ci skip] 2018-03-24 16:27:06 +02:00
Mitchell Krog
4e3079ff1e
Bots (+) Closes: #144
BDCbot
IstellaBot
DnyzBot
AlphaBot
2018-03-24 16:20:52 +02:00
Travis
523fc20c85 V3.2018.03.1068 [ci skip] 2018-03-24 16:18:09 +02:00
Mitchell Krog
6bd44637db
Bots (+)
Shodan
2018-03-24 16:12:16 +02:00
Mitchell Krog
36e141a041
Bots (-) - Closes: #143
RankingBot
RankingBot2
2018-03-24 11:59:28 +02:00
Travis
74de038d5c V3.2018.03.1062 [ci skip] 2018-03-08 10:25:44 +02:00