Jos512
|
9778d57afd
|
Added archive bot
|
2018-06-16 14:27:58 +02:00 |
|
Travis
|
2a3bc40f73
|
V3.2018.06.1113 [ci skip]
|
2018-06-08 10:44:03 +02:00 |
|
Mitchell Krog
|
ec919d888f
|
Bot (+) Quick-Crawler
Referrer (+) scrapinghub.com
Closes: #181
|
2018-06-08 10:28:26 +02:00 |
|
Travis
|
cdc76609d9
|
V3.2018.06.1112 [ci skip]
|
2018-06-07 11:21:22 +02:00 |
|
Mitchell Krog
|
ad5750f32d
|
Bot (+) Referrer (+)
- PXBroker / db.speedup.gdn
Closes: #180
|
2018-06-07 11:12:56 +02:00 |
|
Travis
|
a9ef73b12b
|
V3.2018.06.1109 [ci skip]
|
2018-06-06 16:46:52 +02:00 |
|
Mitchell Krog
|
fe2647c0e6
|
User-Agent + - Mr.4x3
Closes: #177
|
2018-06-06 16:36:40 +02:00 |
|
Mitchell Krog
|
99a41f81e3
|
Remove "NYU" already covered by "scan.lol"
-REF: ab1c59ad02 (commitcomment-29255444)
|
2018-06-06 08:40:45 +02:00 |
|
Mark Beech
|
f5fdf6ec02
|
Update bad-user-agents.list
|
2018-05-16 21:44:27 +01:00 |
|
Travis
|
fd87312e39
|
V3.2018.04.1098 [ci skip]
|
2018-04-21 13:23:48 +02:00 |
|
Mitchell Krog
|
23fb3f62c6
|
Referrers (+) / Bots (+)
|
2018-04-21 13:17:11 +02:00 |
|
Travis
|
5d04fa114d
|
V3.2018.04.1095 [ci skip]
|
2018-04-20 11:23:39 +02:00 |
|
Mitchell Krog
|
b4795ed886
|
Bots (+)
OutclicksBot
|
2018-04-20 11:16:50 +02:00 |
|
Travis
|
9ead97e9d6
|
V3.2018.04.1094 [ci skip]
|
2018-04-20 09:28:10 +02:00 |
|
Mitchell Krog
|
4f83090d8c
|
Bots (+)
Zauba Search for Research; http://www.zauba.io/; admin@zauba.io
- Does not request robots.txt
|
2018-04-20 09:21:32 +02:00 |
|
Travis
|
83596094ae
|
V3.2018.04.1091 [ci skip]
|
2018-04-14 16:58:58 +02:00 |
|
Mitchell Krog
|
5a75d55c5e
|
Merge pull request #153 from StarkRavingZA/master
Request Re-Addition of RankingBot
|
2018-04-14 16:55:41 +02:00 |
|
Travis
|
1611ce1591
|
V3.2018.04.1090 [ci skip]
|
2018-04-13 09:00:19 +02:00 |
|
Mitchell Krog
|
c56c381dd3
|
Bots (+)
IndeedBot
Closes: #155
|
2018-04-13 08:53:37 +02:00 |
|
Travis
|
bae8a933c6
|
V3.2018.04.1082 [ci skip]
|
2018-04-03 09:41:40 +02:00 |
|
Mitchell Krog
|
98623076e4
|
Bots (+)
crawler4j
|
2018-04-03 09:35:30 +02:00 |
|
StarkRavingZA
|
97fa50811e
|
Request Re-Addition of RankingBot
|
2018-03-31 16:47:40 +02:00 |
|
Travis
|
8ab7b68053
|
V3.2018.03.1078 [ci skip]
|
2018-03-31 12:27:12 +02:00 |
|
Mitchell Krog
|
aec2eeaaa8
|
Referrers (+) / Bots (+)
REFERRERS
veselokloun.ru
edudocs.net
exdocsfiles.com
pillscheap24h.com
animenime.ru
sundrugstore.com
seocheckupx.com
game300.ru
hccoder.info
buntube.net
filesclub.net
saneyes.com
compiko.info
bitcoinpile.com
docs4all.com
infodocsportal.com
mixx.com
vzube.com
yun56.co
BOTS
scan.lol
NYU
MauiBot
|
2018-03-31 12:21:10 +02:00 |
|
Travis
|
25fd2ae663
|
V3.2018.03.1074 [ci skip]
|
2018-03-29 09:23:06 +02:00 |
|
Mitchell Krog
|
4341d84d24
|
Bots (+) Closes: #146
crawler - crawler.feedback@gmail.com
|
2018-03-29 09:17:12 +02:00 |
|
Travis
|
4963244255
|
V3.2018.03.1073 [ci skip]
|
2018-03-27 11:40:03 +02:00 |
|
StarkRavingZA
|
fcdb5bf11e
|
Add new User Agents and Referer
|
2018-03-27 11:19:09 +02:00 |
|
Travis
|
f46621d1a2
|
V3.2018.03.1069 [ci skip]
|
2018-03-24 16:27:06 +02:00 |
|
Mitchell Krog
|
4e3079ff1e
|
Bots (+) Closes: #144
BDCbot
IstellaBot
DnyzBot
AlphaBot
|
2018-03-24 16:20:52 +02:00 |
|
Travis
|
523fc20c85
|
V3.2018.03.1068 [ci skip]
|
2018-03-24 16:18:09 +02:00 |
|
Mitchell Krog
|
6bd44637db
|
Bots (+)
Shodan
|
2018-03-24 16:12:16 +02:00 |
|
Mitchell Krog
|
36e141a041
|
Bots (-) - Closes: #143
RankingBot
RankingBot2
|
2018-03-24 11:59:28 +02:00 |
|
Travis
|
74de038d5c
|
V3.2018.03.1062 [ci skip]
|
2018-03-08 10:25:44 +02:00 |
|
Mitchell Krog
|
8f3608add5
|
Bot (+ xpymep1.exe)
|
2018-03-08 10:20:36 +02:00 |
|
Mitchell Krog
|
da76f59535
|
Bots (+/-)
|
2018-02-28 09:49:17 +02:00 |
|
Travis
|
76b6659c4a
|
V3.2018.01.1042 [ci skip]
|
2018-01-31 09:23:31 +02:00 |
|
Mitchell Krog
|
b1fc051b76
|
User-Agent (+)
|
2018-01-31 09:19:20 +02:00 |
|
Mitchell Krog
|
dee847d8ca
|
FIX BROKEN BOTS Fix: #115
|
2017-12-20 13:03:51 +02:00 |
|
Travis
|
435952675e
|
V3.2017.12.1013 [ci skip]
|
2017-12-20 11:30:31 +02:00 |
|
Mitchell Krog
|
db27cb6378
|
Update Bad Bots
|
2017-12-20 11:27:38 +02:00 |
|
Travis
|
fcf42f1805
|
V3.2017.11.998 [ci skip]
|
2017-11-18 14:57:38 +02:00 |
|
Mitchell Krog
|
c766c67290
|
Bots (+)
|
2017-11-18 13:13:49 +02:00 |
|
Mitchell Krog
|
6b896756ef
|
UPDATE Bad Bots Fix: #110
|
2017-11-15 15:28:50 +02:00 |
|
Mitchell Krog
|
ff17386fff
|
UPDATE User Agents / Bots
|
2017-11-14 10:18:31 +02:00 |
|
Mitchell Krog
|
8a7c9a4bce
|
UPDATE Bad Bots
|
2017-11-13 11:30:16 +02:00 |
|
Mitchell Krog
|
59391a1191
|
UPDATE Bad Bots / Corrections in Naming of Bots for Exact Matching with Regex
|
2017-11-13 10:04:11 +02:00 |
|
Travis
|
86b286a7e9
|
V3.2017.11.988 [ci skip]
|
2017-11-13 09:23:52 +02:00 |
|
Mitchell Krog
|
d0734a2540
|
UPDATE Bad Bots
|
2017-11-13 09:20:19 +02:00 |
|
Travis
|
5c924376cd
|
V3.2017.11.987 [ci skip]
|
2017-11-12 11:51:59 +02:00 |
|