vrr/nDPI
2
0
Fork 0
mirror of https://github.com/vel21ripn/nDPI.git synced 2026-04-28 06:59:40 +00:00
Code Issues Projects Releases Packages Wiki Activity Actions 5

Updated bots and scanners list

Browse source 
Improved lists file parsing
This commit is contained in:
Luca Deri 2025-05-29 10:38:24 +02:00
parent bebacc853d
commit c36e4bd5de
6 changed files with 398 additions and 12804 deletions
Download patch file Download diff file Expand all files Collapse all files

12794
lists/123_bots.list
View file

File diff suppressed because it is too large Load diff

382
lists/124_bots.list Normal file
View file

@ -0,0 +1,382 @@
#
# https://github.com/AnTheMaker/GoodBots/tree/main
# bingbot.ips googlebot.ips yandex.ips webpagetestbot.ips
#
157.55.39.0/24
207.46.13.0/24
40.77.167.0/24
13.66.139.0/24
13.66.144.0/24
52.167.144.0/24
13.67.10.16/28
13.69.66.240/28
13.71.172.224/28
139.217.52.0/28
191.233.204.224/28
20.36.108.32/28
20.43.120.16/28
40.79.131.208/28
40.79.186.176/28
52.231.148.0/28
20.79.107.240/28
51.105.67.0/28
20.125.163.80/28
40.77.188.0/22
65.55.210.0/24
199.30.24.0/23
40.77.202.0/24
40.77.139.0/25
20.74.197.0/28
20.15.133.160/27
40.77.177.0/24
40.77.178.0/23
2001:4860:4801:10::/64
2001:4860:4801:11::/64
2001:4860:4801:12::/64
2001:4860:4801:13::/64
2001:4860:4801:14::/64
2001:4860:4801:15::/64
2001:4860:4801:16::/64
2001:4860:4801:17::/64
2001:4860:4801:18::/64
2001:4860:4801:19::/64
2001:4860:4801:1a::/64
2001:4860:4801:1b::/64
2001:4860:4801:1c::/64
2001:4860:4801:1d::/64
2001:4860:4801:1e::/64
2001:4860:4801:1f::/64
2001:4860:4801:20::/64
2001:4860:4801:21::/64
2001:4860:4801:22::/64
2001:4860:4801:23::/64
2001:4860:4801:24::/64
2001:4860:4801:25::/64
2001:4860:4801:26::/64
2001:4860:4801:27::/64
2001:4860:4801:28::/64
2001:4860:4801:29::/64
2001:4860:4801:2::/64
2001:4860:4801:2a::/64
2001:4860:4801:2b::/64
2001:4860:4801:2c::/64
2001:4860:4801:2d::/64
2001:4860:4801:2e::/64
2001:4860:4801:2f::/64
2001:4860:4801:30::/64
2001:4860:4801:31::/64
2001:4860:4801:32::/64
2001:4860:4801:33::/64
2001:4860:4801:34::/64
2001:4860:4801:35::/64
2001:4860:4801:36::/64
2001:4860:4801:37::/64
2001:4860:4801:38::/64
2001:4860:4801:39::/64
2001:4860:4801:3a::/64
2001:4860:4801:3b::/64
2001:4860:4801:3c::/64
2001:4860:4801:3d::/64
2001:4860:4801:3e::/64
2001:4860:4801:3f::/64
2001:4860:4801:40::/64
2001:4860:4801:41::/64
2001:4860:4801:42::/64
2001:4860:4801:43::/64
2001:4860:4801:44::/64
2001:4860:4801:45::/64
2001:4860:4801:46::/64
2001:4860:4801:47::/64
2001:4860:4801:48::/64
2001:4860:4801:49::/64
2001:4860:4801:4a::/64
2001:4860:4801:4b::/64
2001:4860:4801:4c::/64
2001:4860:4801:4d::/64
2001:4860:4801:50::/64
2001:4860:4801:51::/64
2001:4860:4801:52::/64
2001:4860:4801:53::/64
2001:4860:4801:54::/64
2001:4860:4801:55::/64
2001:4860:4801:56::/64
2001:4860:4801:57::/64
2001:4860:4801:60::/64
2001:4860:4801:61::/64
2001:4860:4801:62::/64
2001:4860:4801:63::/64
2001:4860:4801:64::/64
2001:4860:4801:65::/64
2001:4860:4801:66::/64
2001:4860:4801:67::/64
2001:4860:4801:68::/64
2001:4860:4801:69::/64
2001:4860:4801:6a::/64
2001:4860:4801:6b::/64
2001:4860:4801:6c::/64
2001:4860:4801:6d::/64
2001:4860:4801:6e::/64
2001:4860:4801:6f::/64
2001:4860:4801:70::/64
2001:4860:4801:71::/64
2001:4860:4801:72::/64
2001:4860:4801:73::/64
2001:4860:4801:74::/64
2001:4860:4801:75::/64
2001:4860:4801:76::/64
2001:4860:4801:77::/64
2001:4860:4801:78::/64
2001:4860:4801:79::/64
2001:4860:4801:7a::/64
2001:4860:4801:7b::/64
2001:4860:4801:80::/64
2001:4860:4801:81::/64
2001:4860:4801:82::/64
2001:4860:4801:83::/64
2001:4860:4801:84::/64
2001:4860:4801:85::/64
2001:4860:4801:86::/64
2001:4860:4801:87::/64
2001:4860:4801:88::/64
2001:4860:4801:90::/64
2001:4860:4801:91::/64
2001:4860:4801:92::/64
2001:4860:4801:93::/64
2001:4860:4801:94::/64
2001:4860:4801:95::/64
2001:4860:4801:96::/64
2001:4860:4801:97::/64
2001:4860:4801:a0::/64
2001:4860:4801:a1::/64
2001:4860:4801:a2::/64
2001:4860:4801:a3::/64
2001:4860:4801:a4::/64
2001:4860:4801:a5::/64
2001:4860:4801:a6::/64
2001:4860:4801:a7::/64
2001:4860:4801:a8::/64
2001:4860:4801:a9::/64
2001:4860:4801:aa::/64
2001:4860:4801:ab::/64
2001:4860:4801:ac::/64
2001:4860:4801:b0::/64
2001:4860:4801:b1::/64
2001:4860:4801:b2::/64
2001:4860:4801:b3::/64
2001:4860:4801:b4::/64
2001:4860:4801:c::/64
2001:4860:4801:f::/64
192.178.4.0/27
192.178.4.128/27
192.178.4.160/27
192.178.4.32/27
192.178.4.64/27
192.178.4.96/27
192.178.5.0/27
192.178.6.0/27
192.178.6.128/27
192.178.6.160/27
192.178.6.192/27
192.178.6.224/27
192.178.6.32/27
192.178.6.64/27
192.178.6.96/27
192.178.7.0/27
192.178.7.128/27
192.178.7.160/27
192.178.7.32/27
192.178.7.64/27
192.178.7.96/27
34.100.182.96/28
34.101.50.144/28
34.118.254.0/28
34.118.66.0/28
34.126.178.96/28
34.146.150.144/28
34.147.110.144/28
34.151.74.144/28
34.152.50.64/28
34.154.114.144/28
34.155.98.32/28
34.165.18.176/28
34.175.160.64/28
34.176.130.16/28
34.22.85.0/27
34.64.82.64/28
34.65.242.112/28
34.80.50.80/28
34.88.194.0/28
34.89.10.80/28
34.89.198.80/28
34.96.162.48/28
35.247.243.240/28
66.249.64.0/27
66.249.64.128/27
66.249.64.160/27
66.249.64.192/27
66.249.64.224/27
66.249.64.32/27
66.249.64.64/27
66.249.64.96/27
66.249.65.0/27
66.249.65.128/27
66.249.65.160/27
66.249.65.192/27
66.249.65.224/27
66.249.65.32/27
66.249.65.64/27
66.249.65.96/27
66.249.66.0/27
66.249.66.128/27
66.249.66.160/27
66.249.66.192/27
66.249.66.224/27
66.249.66.32/27
66.249.66.64/27
66.249.66.96/27
66.249.67.0/27
66.249.68.0/27
66.249.68.128/27
66.249.68.160/27
66.249.68.32/27
66.249.68.64/27
66.249.68.96/27
66.249.69.0/27
66.249.69.128/27
66.249.69.160/27
66.249.69.192/27
66.249.69.224/27
66.249.69.32/27
66.249.69.64/27
66.249.69.96/27
66.249.70.0/27
66.249.70.128/27
66.249.70.160/27
66.249.70.192/27
66.249.70.224/27
66.249.70.32/27
66.249.70.64/27
66.249.70.96/27
66.249.71.0/27
66.249.71.128/27
66.249.71.160/27
66.249.71.192/27
66.249.71.224/27
66.249.71.32/27
66.249.71.64/27
66.249.71.96/27
66.249.72.0/27
66.249.72.128/27
66.249.72.160/27
66.249.72.192/27
66.249.72.224/27
66.249.72.32/27
66.249.72.64/27
66.249.72.96/27
66.249.73.0/27
66.249.73.128/27
66.249.73.160/27
66.249.73.192/27
66.249.73.224/27
66.249.73.32/27
66.249.73.64/27
66.249.73.96/27
66.249.74.0/27
66.249.74.128/27
66.249.74.160/27
66.249.74.192/27
66.249.74.224/27
66.249.74.32/27
66.249.74.64/27
66.249.74.96/27
66.249.75.0/27
66.249.75.128/27
66.249.75.160/27
66.249.75.192/27
66.249.75.224/27
66.249.75.32/27
66.249.75.64/27
66.249.75.96/27
66.249.76.0/27
66.249.76.128/27
66.249.76.160/27
66.249.76.192/27
66.249.76.224/27
66.249.76.32/27
66.249.76.64/27
66.249.76.96/27
66.249.77.0/27
66.249.77.128/27
66.249.77.160/27
66.249.77.192/27
66.249.77.224/27
66.249.77.32/27
66.249.77.64/27
66.249.77.96/27
66.249.78.0/27
66.249.78.32/27
66.249.78.64/27
66.249.78.96/27
66.249.79.0/27
66.249.79.128/27
66.249.79.160/27
66.249.79.192/27
66.249.79.224/27
66.249.79.32/27
66.249.79.64/27
66.249.79.96/27
2a02:6b8::/29
5.45.192.0/18
5.255.192.0/18
37.9.64.0/18
37.140.128.0/18
77.88.0.0/18
84.252.160.0/19
87.250.224.0/19
90.156.176.0/22
93.158.128.0/18
95.108.128.0/17
141.8.128.0/18
178.154.128.0/18
213.180.192.0/19
185.32.187.0/2434.21.9.50
34.106.208.213
34.94.159.140
34.130.107.20
34.39.131.22
34.240.49.81
35.242.177.6
13.36.154.207
34.91.238.70
34.159.56.80
34.154.170.5
35.228.243.201
13.246.114.251
15.184.48.78
20.74.211.96
43.204.105.75
34.124.137.169
150.109.170.94
150.109.181.129
150.109.172.224
124.156.244.42
34.101.194.174
34.92.96.4
1.117.61.116
115.159.57.46
124.220.31.125
150.158.3.127
49.232.162.18
154.8.217.128
49.233.249.157
101.43.149.237
104.199.223.119
39.104.13.61
43.200.31.191
34.84.123.12
13.208.89.86
34.40.137.12
13.70.181.210
172.70.215.14
162.158.186.123
162.158.90.134

0
lists/124_scanners.list → lists/125_scanners.list
View file

22
src/lib/ndpi_main.c
View file

@ -5224,19 +5224,23 @@ int load_category_file_fd(struct ndpi_detection_module_struct *ndpi_str,
line[i] = '\0';
break;
}
if (line[i] != '-' && line[i] != '.' && ndpi_isalnum(line[i]) == 0
if (line[i] != '-'
&& line[i] != '.'
&& line[i] != ':'
&& line[i] != '/'
&& ndpi_isalnum(line[i]) == 0
/* non standard checks for the sake of compatibility */
&& line[i] != '_')
break;
}
if (i != len - 2 && i != len - 1)
{
NDPI_LOG_ERR(ndpi_str, "[NDPI] Failed to read file line #%u, invalid characters found\n",
lines_read);
failed_lines++;
continue;
}
if ((i != len - 2) && (i != len - 1) && (line[i] != '\0')) {
NDPI_LOG_ERR(ndpi_str, "[NDPI] Failed to read file line #%u [%s], invalid characters [%c] found [pos: %u]\n",
lines_read, line, line[i], i);
failed_lines++;
continue;
}
if(ndpi_load_category(ndpi_str, line, category_id, NULL) >= 0)
num_loaded++;
@ -5244,6 +5248,7 @@ int load_category_file_fd(struct ndpi_detection_module_struct *ndpi_str,
if(failed_lines)
return(-1 * failed_lines);
return(num_loaded);
}
@ -8602,6 +8607,7 @@ int ndpi_load_ip_category(struct ndpi_detection_module_struct *ndpi_str,
NDPI_LOG_DBG2(ndpi_str, "Invalid ip4/ip4+netmask: %s\n", ip_address_and_mask);
return(-1);
}
node = add_to_ptree(ndpi_str->custom_categories.ipAddresses_shadow, AF_INET, &pin, bits);
} else if(is_ipv6 && ndpi_str->custom_categories.ipAddresses6_shadow) {
struct in6_addr pin6;

2
tests/cfgs/default/result/bot.pcap.out
View file

@ -24,4 +24,4 @@ HTTP 402 431124 1
Acceptable 402 431124 1
1 TCP 40.77.167.36:64768 <-> 89.31.72.220:80 [VLAN: 77][proto: 7/HTTP][IP: 276/Azure][ClearText][Confidence: DPI][FPC: 276/Azure, Confidence: IP address][DPI packets: 6][cat: Web/5][115 pkts/7672 bytes <-> 287 pkts/423452 bytes][Goodput ratio: 4/96][5.66 sec][Hostname/SNI: atlanteditorino.it][bytes ratio: -0.964 (Download)][IAT c2s/s2c min/avg/max/stddev: 0/0 58/3 4532/106 489/16][Pkt Len c2s/s2c min/avg/max/stddev: 64/64 67/1475 374/1498 29/171][URL: atlanteditorino.it/quartieri/img/S.Donato_M.Vittoria1930_B.jpg][StatusCode: 200][Content-Type: image/jpeg][Server: Apache][User-Agent: Mozilla/5.0 (compatible; bingbot/2.0; +http://www.bing.com/bingbot.htm)][Risk: ** Crawler/Bot **][Risk Score: 10][Risk Info: UA Mozilla/5.0 (compatible; bingbot/2.0; +http://www.bing.com/b][TCP Fingerprint: 194_128_64240_f65176b9e3ec/Unknown][PLAIN TEXT (GET /quartieri/im)][Plen Bins: 0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,100,0,0]
1 TCP 40.77.167.36:64768 <-> 89.31.72.220:80 [VLAN: 77][proto: 7/HTTP][IP: 276/Azure][ClearText][Confidence: DPI][FPC: 276/Azure, Confidence: IP address][DPI packets: 6][cat: Bots/124][115 pkts/7672 bytes <-> 287 pkts/423452 bytes][Goodput ratio: 4/96][5.66 sec][Hostname/SNI: atlanteditorino.it][bytes ratio: -0.964 (Download)][IAT c2s/s2c min/avg/max/stddev: 0/0 58/3 4532/106 489/16][Pkt Len c2s/s2c min/avg/max/stddev: 64/64 67/1475 374/1498 29/171][URL: atlanteditorino.it/quartieri/img/S.Donato_M.Vittoria1930_B.jpg][StatusCode: 200][Content-Type: image/jpeg][Server: Apache][User-Agent: Mozilla/5.0 (compatible; bingbot/2.0; +http://www.bing.com/bingbot.htm)][Risk: ** Crawler/Bot **][Risk Score: 10][Risk Info: UA Mozilla/5.0 (compatible; bingbot/2.0; +http://www.bing.com/b][TCP Fingerprint: 194_128_64240_f65176b9e3ec/Unknown][PLAIN TEXT (GET /quartieri/im)][Plen Bins: 0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,100,0,0]

2
tests/cfgs/disable_use_client_ip/result/bot.pcap.out
View file

@ -24,4 +24,4 @@ HTTP 402 431124 1
Acceptable 402 431124 1
1 TCP 40.77.167.36:64768 <-> 89.31.72.220:80 [VLAN: 77][proto: 7/HTTP][IP: 0/Unknown][ClearText][Confidence: DPI][FPC: 0/Unknown, Confidence: Unknown][DPI packets: 6][cat: Web/5][115 pkts/7672 bytes <-> 287 pkts/423452 bytes][Goodput ratio: 4/96][5.66 sec][Hostname/SNI: atlanteditorino.it][bytes ratio: -0.964 (Download)][IAT c2s/s2c min/avg/max/stddev: 0/0 58/3 4532/106 489/16][Pkt Len c2s/s2c min/avg/max/stddev: 64/64 67/1475 374/1498 29/171][URL: atlanteditorino.it/quartieri/img/S.Donato_M.Vittoria1930_B.jpg][StatusCode: 200][Content-Type: image/jpeg][Server: Apache][User-Agent: Mozilla/5.0 (compatible; bingbot/2.0; +http://www.bing.com/bingbot.htm)][Risk: ** Crawler/Bot **][Risk Score: 10][Risk Info: UA Mozilla/5.0 (compatible; bingbot/2.0; +http://www.bing.com/b][TCP Fingerprint: 194_128_64240_f65176b9e3ec/Unknown][PLAIN TEXT (GET /quartieri/im)][Plen Bins: 0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,100,0,0]
1 TCP 40.77.167.36:64768 <-> 89.31.72.220:80 [VLAN: 77][proto: 7/HTTP][IP: 0/Unknown][ClearText][Confidence: DPI][FPC: 0/Unknown, Confidence: Unknown][DPI packets: 6][cat: Bots/124][115 pkts/7672 bytes <-> 287 pkts/423452 bytes][Goodput ratio: 4/96][5.66 sec][Hostname/SNI: atlanteditorino.it][bytes ratio: -0.964 (Download)][IAT c2s/s2c min/avg/max/stddev: 0/0 58/3 4532/106 489/16][Pkt Len c2s/s2c min/avg/max/stddev: 64/64 67/1475 374/1498 29/171][URL: atlanteditorino.it/quartieri/img/S.Donato_M.Vittoria1930_B.jpg][StatusCode: 200][Content-Type: image/jpeg][Server: Apache][User-Agent: Mozilla/5.0 (compatible; bingbot/2.0; +http://www.bing.com/bingbot.htm)][Risk: ** Crawler/Bot **][Risk Score: 10][Risk Info: UA Mozilla/5.0 (compatible; bingbot/2.0; +http://www.bing.com/b][TCP Fingerprint: 194_128_64240_f65176b9e3ec/Unknown][PLAIN TEXT (GET /quartieri/im)][Plen Bins: 0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,100,0,0]