Commit graph

351 commits

Author SHA1 Message Date
Luca Deri
76829b413f Implements support for symbolic host names (#2123) 2023-10-29 22:54:45 +01:00
Ivan Nardi
03fd155ae3
IPv6: add support for custom categories (#2126) 2023-10-29 12:56:44 +01:00
Ivan Nardi
32b50f5aa4
IPv6: add support for IPv6 risk exceptions (#2122) 2023-10-29 12:14:20 +01:00
Ivan Nardi
c711251578
IPv6: add support for custom rules (#2120) 2023-10-29 11:26:35 +01:00
Ivan Nardi
e8e4b9e8ff
IPv6: add support for IPv6 risk tree (#2118)
Fix the script to download crawler addressess
2023-10-27 13:58:15 +02:00
Toni
ed17f4d658
Improved Protobuf dissector. (#2119)
* tag extraction/validation was done wrong

Signed-off-by: Toni Uhlig <matzeton@googlemail.com>
2023-10-27 10:19:47 +02:00
Ivan Nardi
8b07be4b9f
Jabber: remove support for UDP (#2115)
Jabber/XMPP is only over TCP (even the name `ndpi_search_jabber_tcp`
suggests that...).

Bug introduced in 5266c726f
2023-10-26 20:16:27 +02:00
Ivan Nardi
611c3b66f0
ipv6: add support for ipv6 addresses lists (#2113) 2023-10-26 20:15:44 +02:00
Maatuq
4a8e7105b2
add ethereum protocol dissector. (#2111)
as explained here for bitcoin https://www.ntop.org/guides/nDPI/protocols.html#ndpi-protocol-bitcoin
the same is applicable for ethereum.
ethereum detection was removed from mining protocol and is now handled separately.

Signed-off-by: Mahmoud Maatuq <mahmoudmatook.mm@gmail.com>
2023-10-25 12:44:33 +02:00
Toni
e70333de87
Added generic Google Protobuf dissector. (#2109)
Signed-off-by: Toni Uhlig <matzeton@googlemail.com>
2023-10-24 12:18:31 +02:00
Toni Uhlig
a443bba0dd Add CAN over Ethernet dissector.
Signed-off-by: Toni Uhlig <matzeton@googlemail.com>
2023-10-23 13:45:56 +02:00
Toni Uhlig
25c54dd6d7 Improved CryNetwork protocol dissector.
Signed-off-by: Toni Uhlig <matzeton@googlemail.com>
2023-10-23 13:44:34 +02:00
Toni Uhlig
f69909d49b Add Remote Management Control Protocol (RMCP).
Signed-off-by: Toni Uhlig <matzeton@googlemail.com>
2023-10-19 19:50:57 +02:00
Toni
01f384f7ff
Improved Steam detection by adding steamdiscover pattern. (#2105)
Signed-off-by: Toni Uhlig <matzeton@googlemail.com>
2023-10-17 22:13:02 +02:00
Luca Deri
b6179467eb Added NDPI_MALWARE_HOST_CONTACTED flow risk 2023-10-13 00:01:55 +02:00
lns
304747f1fa Improved MGCP detection by allowing '\r' as line feed.
Signed-off-by: lns <matzeton@googlemail.com>
Signed-off-by: Toni Uhlig <matzeton@googlemail.com>
2023-10-11 16:44:26 +02:00
Nardi Ivan
4a0eda69ad QUIC: export QUIC version as metadata 2023-10-11 15:15:20 +02:00
Toni
a98d7ff433
Added HAProxy protocol. (#2088)
* fixed tests/do.sh.in failure print

Signed-off-by: lns <matzeton@googlemail.com>
Signed-off-by: Toni Uhlig <matzeton@googlemail.com>
2023-10-02 18:10:47 +02:00
Luca
77e5daf03e Cleaned up mining datastructure 2023-09-27 17:05:12 +02:00
Luca Deri
725fcf4852 Renamed HTTP/2 to HTTP2 as the '/' can have side effects with applications sitting on top of nDPI 2023-09-20 11:17:09 +02:00
Ivan Nardi
6925890383
Add support for (un-encrypted) HTTP/2 (#2087)
Plaintext HTTP/2 is quite rare on the general "internet" but it is
used in some private networks (example: 5G core network)
2023-09-18 14:06:09 +02:00
Thomas Winter
5bcf3c2ddd tftp: update pcap results
The two malformed TFTP packets are no longer considered as risk
and instead match by port only.
This is because the TFTP detection was rather sparse so could
match on several other protocols if the first two opcode bytes
happened to match.
2023-09-12 13:12:14 +02:00
Luca Deri
1bf7e5face Fixes matches with domain name strings that start with a dot 2023-09-11 22:50:03 +02:00
Ivan Nardi
0aa1cf7245
Update every ip lists (#2079) 2023-09-10 12:13:20 +02:00
Ivan Nardi
2b883b93be
Fix some errors found by fuzzers (#2078)
Fix compilation on Windows.
"dirent.h" file has been taken from https://github.com/tronkko/dirent/

Fix Python bindings

Fix some warnings with x86_64-w64-mingw32-gcc:
```
protocols/dns.c: In function ‘ndpi_search_dns’:
protocols/dns.c:775:41: error: cast from pointer to integer of different size [-Werror=pointer-to-int-cast]
  775 |       unsigned long first_element_len = (unsigned long)dot - (unsigned long)_hostname;
      |                                         ^
protocols/dns.c:775:62: error: cast from pointer to integer of different size [-Werror=pointer-to-int-cast]
  775 |       unsigned long first_element_len = (unsigned long)dot - (unsigned long)_hostname;
      |
```
```
In file included from ndpi_bitmap64.c:31:
third_party/include/binaryfusefilter.h: In function ‘binary_fuse8_hash’:
third_party/include/binaryfusefilter.h:160:32: error: left shift count >= width of type [-Werror=shift-count-overflow]
  160 |     uint64_t hh = hash & ((1UL << 36) - 1);
```
```
In function ‘ndpi_match_custom_category’,
    inlined from ‘ndpi_fill_protocol_category.part.0’ at ndpi_main.c:7056:16:
ndpi_main.c:3419:3: error: ‘strncpy’ specified bound depends on the length of the source argument [-Werror=stringop-overflow=]
 3419 |   strncpy(buf, name, name_len);
```
2023-09-10 11:09:59 +02:00
Luca Deri
63977459b6 Added OperaVPN detection 2023-09-09 18:17:26 +02:00
Luca Deri
076edeab54 Enhance DNS risk for long hostnames (> 32) 2023-09-09 18:09:57 +02:00
Luca Deri
f50a4d7e85 Improved detection of invalid chars in DNS names 2023-09-09 17:46:13 +02:00
Luca Deri
770e460ba9 Added NDPI_TLS_ALPN_SNI_MISMATCH flow risk 2023-09-07 23:42:42 +02:00
Luca Deri
978df906b3 Improved classification further reducing memory used 2023-09-05 17:03:20 +02:00
Luca Deri
36abf06c6f Swap from Aho-Corasick to an experimental/home-grown algorithm that uses a probabilistic
approach for handling Internet domain names.

For switching back to Aho-Corasick it is necessary to edit
ndpi-typedefs.h and uncomment the line
// #define USE_LEGACY_AHO_CORASICK

[1] With Aho-Corasick
$ ./example/ndpiReader -G ./lists/ -i tests/pcap/ookla.pcap | grep Memory
nDPI Memory statistics:
nDPI Memory (once):      37.34 KB
Flow Memory (per flow):  960 B
Actual Memory:           33.09 MB
Peak Memory:             33.09 MB

[2] With the new algorithm
$ ./example/ndpiReader -G ./lists/ -i tests/pcap/ookla.pcap | grep Memory
nDPI Memory statistics:
nDPI Memory (once):      37.31 KB
Flow Memory (per flow):  960 B
Actual Memory:           7.42 MB
Peak Memory:             7.42 MB

In essence from ~33 MB to ~7 MB

This new algorithm will enable larger lists to be loaded (e.g. top 1M domans
https://s3-us-west-1.amazonaws.com/umbrella-static/index.html)

In ./lists there are file names that are named as <category>_<string>.list
With -G ndpiReader can load all of them at startup
2023-08-29 17:34:04 +02:00
Ivan Nardi
cc4461f424
fuzz: extend coverage (#2073) 2023-08-20 15:18:19 +02:00
snicket2100
1fbe8a2385
Mullvad VPN service added (based on entry node IP addresses) (#2062) 2023-08-02 19:44:16 +02:00
Toni
e4d3d619bc
Add Service Location Protocol dissector. (#2036)
Signed-off-by: Toni Uhlig <matzeton@googlemail.com>
2023-08-01 08:50:46 +02:00
Ivan Nardi
5019022e13
DNS: extract geolocation information, if available (#2065)
The option NSID (RFC5001) is used by Google DNS to report the
airport code of the metro where the DNS query is handled.

This option is quite rare, but the added overhead in DNS code is pretty
much zero for "normal" DNS traffic
2023-07-31 07:44:43 +02:00
Ivan Nardi
bc91192aca
ProtonVPN: split the ip list (#2060)
Use two separate lists:
* one for the ingress nodes, which triggers a ProtonVPN classification
* one for the egress nodes, which triggers the
`NDPI_ANONYMOUS_SUBSCRIBER` risk

Add a command line option (to `ndpiReader`) to easily test IP/port
matching.

Add another example of custom rule.
2023-07-27 09:05:22 +02:00
Ivan Nardi
3326fa258e
Add an heuristic to detect fully encrypted flows (#2058)
A fully encrypted session is a flow where every bytes of the
payload is encrypted in an attempt to “look like nothing”.
The heuristic needs only the very first packet of the flow.
See: https://www.usenix.org/system/files/sec23fall-prepub-234-wu-mingshi.pdf

A basic, but generic, inplementation of the popcpunt alg has been added
2023-07-26 09:09:12 +02:00
Ivan Nardi
7a1509bf60
zabbix: improve detection (#2055) 2023-07-21 03:42:36 +02:00
Ivan Nardi
c85f2fb0f4
TLS: add basic, basic, detection of Encrypted ClientHello (#2053) 2023-07-21 03:41:43 +02:00
Ivan Nardi
fa0bd515b5
Add detection of Roblox games (#2054) 2023-07-21 03:39:40 +02:00
Ivan Nardi
3edfad01a1
fuzz: extend fuzzing coverage (#2052)
Added/merged some traces.
Improved Socks identification
2023-07-18 07:41:56 +02:00
Ivan Nardi
2bbde5bad3
Fix compilation in CI jobs (#2048) 2023-07-15 14:26:51 +02:00
Ivan Nardi
890f17788b
ndpireader: fix detection of DoH traffic based on packet distributions (#2045) 2023-07-14 23:20:06 +02:00
Luca Deri
8e960f033d Adds new pcap for testing "funny" HTTP servers 2023-07-14 21:12:11 +02:00
Luca Deri
fea09e825b Fixes risk mask exception handling while improving the overall performance 2023-07-14 19:52:34 +02:00
Ivan Nardi
0c5a17accb
RDP: improve detection over UDP (#2043) 2023-07-13 10:50:13 +02:00
Chiara Maggi
0b0f255cc2
added feature to extract filename from http attachment (#2037)
* added feature to extract filename from http attachment

* fixed some issues

* added check for filename format

* added check for filename format

* remove an unnecessary print

* changed the size from 952 to 960

* modified some test result files

* small changes string size

* comment removed and mallocs checked
2023-07-11 22:45:19 +02:00
Ivan Nardi
950f5cc4e3
fuzz: extend fuzzing coverage (#2040)
Some notes:
* libinjection: according to https://github.com/libinjection/libinjection/issues/44,
it seems NULL characters are valid in the input string;
* RTP: `rtp_get_stream_type()` is called only for RTP packets; if you
want to tell RTP from RTCP you should use `is_rtp_or_rtcp()`;
* TLS: unnecessary check; we already make the same check just above, at
the beginning of the `while` loop
2023-07-11 10:12:08 +02:00
Ivan Nardi
bb978907e6
STUN: fix detection of Google Voip apps (#2031)
Fix: 2c7fb9179
2023-07-05 14:37:38 +02:00
Ivan Nardi
7ffd31ebc3
STUN: avoid FacebookVoip false positives (#2029)
Attribute 0xC057 is defined in the Google public implementation of
webrtc (which is used by Google products but also by other applications)
2023-07-03 17:21:46 +02:00