Commit graph

68 commits

Author SHA1 Message Date
Ivan Nardi
1796a1d814
LINE_CALL: add detection of LINE voip calls (#1761)
These flows are classifed as `LINE_CALL`; another option was
`RTP/LINE_CALL`. No sure about the best solution...

Extend LINE domains list.

Remove RTP dead code.
2022-10-06 17:09:26 +02:00
Nardi Ivan
ac7a3669da Fix value of ndpi_protocol->protocol_by_ip
Fix: a7c2734b
2022-09-25 13:31:57 +02:00
Nardi Ivan
1f345b311f Sizes of LRU caches are now configurable
0 as size value disable the cache.

The diffs in unit tests are due to the fact that some lookups are
performed before the first insert: before this change these lookups
weren't counted because the cache was not yet initialized, now they are.
2022-09-23 18:33:48 +02:00
Ivan Nardi
a7c2734b38
Remove classification "by-ip" from protocol stack (#1743)
Basically:
* "classification by-ip" (i.e. `flow->guessed_protocol_id_by_ip` is
NEVER returned in the protocol stack (i.e.
`flow->detected_protocol_stack[]`);
* if the application is interested into such information, it can access
`ndpi_protocol->protocol_by_ip` itself.

There are mainly 4 points in the code that set the "classification
by-ip" in the protocol stack:  the generic `ndpi_set_detected_protocol()`/
`ndpi_detection_giveup()` functions and the HTTP/STUN  dissectors.

In the unit tests output, a print about `ndpi_protocol->protocol_by_ip`
has been added for each flow: the huge diff of this commit is mainly due
to that.

Strictly speaking, this change is NOT an API/ABI breakage, but there are
important differences in the classification results. For examples:
* TLS flows without the initial handshake (or without a matching
SNI/certificate) are simply classified as `TLS`;
* similar for HTTP or QUIC flows;
* DNS flows without a matching request domain are simply classified as
`DNS`; we don't have `DNS/Google` anymore just because the server is
8.8.8.8 (that was an outrageous behaviour...);
* flows previusoly classified only "by-ip" are now classified as
`NDPI_PROTOCOL_UNKNOWN`.

See #1425 for other examples of why adding the "classification by-ip" in
the protocol stack is a bad idea.

Please, note that IPV6 is not supported :(  (long standing issue in nDPI) i.e.
`ndpi_protocol->protocol_by_ip` wil be always `NDPI_PROTOCOL_UNKNOWN` for
IPv6 flows.

Define `NDPI_CONFIDENCE_MATCH_BY_IP` has been removed.

Close #1687
2022-09-20 22:24:47 +02:00
Toni Uhlig
a966d37a21 Add CryNetwork dissector.
Signed-off-by: Toni Uhlig <matzeton@googlemail.com>
2022-09-16 16:48:43 +02:00
Toni Uhlig
ec1981c20c Add Syncthing dissector.
Signed-off-by: Toni Uhlig <matzeton@googlemail.com>
2022-09-16 15:21:49 +02:00
Toni
de020b1747
Add NATPMP dissector. (#1738)
Signed-off-by: lns <matzeton@googlemail.com>
Signed-off-by: Toni Uhlig <matzeton@googlemail.com>

Signed-off-by: lns <matzeton@googlemail.com>
Signed-off-by: Toni Uhlig <matzeton@googlemail.com>
2022-09-16 13:32:30 +02:00
Ivan Nardi
4f584f78a0
Fix ndpi_do_guess() (#1731)
Avoid a double call of `ndpi_guess_host_protocol_id()`.
Some code paths work for ipv4/6 both
Remove some never used code.
2022-09-12 19:28:41 +02:00
Toni
2e25c36396
Add TiVoConnect dissector. Fixes #1697. (#1699)
* added static assert if supported, to complain if the flow struct changes

Signed-off-by: lns <matzeton@googlemail.com>
2022-08-08 19:04:20 +02:00
Toni
ac24b35b1f
Add Discord dissector. (#1694)
* fixed RiotGames false positive

Signed-off-by: lns <matzeton@googlemail.com>
2022-08-03 12:03:36 +02:00
Toni
26aafd931c
Add Activision dissector. (#1693)
Signed-off-by: lns <matzeton@googlemail.com>
2022-08-02 16:15:07 +02:00
Toni
ed4f106a0d
Add Softether dissector. (#1679)
Signed-off-by: lns <matzeton@googlemail.com>
2022-07-29 19:29:54 +02:00
Ivan Nardi
405a52ed65
Patricia tree, Ahocarasick automa, LRU cache: add statistics (#1683)
Add (basic) internal stats to the main data structures used by the
library; they might be usefull to check how effective these structures
are.

Add an option to `ndpiReader` to dump them; enabled by default in the
unit tests.
This new option enables/disables dumping of "num dissectors calls"
values, too (see b4cb14ec).
2022-07-29 15:25:00 +02:00
Nardi Ivan
d66aa49787 DTLS: fix exclusion of DTLS protocol
Add an helper to exclude a generic protocol
2022-07-20 19:16:18 +02:00
Ivan Nardi
b4cb14ec19
Keep track of how many dissectors calls we made for each flow (#1657) 2022-07-11 09:47:47 +02:00
Luca Deri
ab09b8ce2e Added unidirectional traffic flow risk 2022-06-20 00:22:13 +02:00
Ivan Nardi
71636dcafd
Sync unit tests results (#1533) 2022-04-27 18:22:11 +02:00
Ivan Nardi
86b97ffb73
Sync unit tests results (#1423)
Fix: 7a3aa41a
2022-01-28 17:17:39 +01:00
Ivan Nardi
3a087e951d
Add a "confidence" field about the reliability of the classification. (#1395)
As a general rule, the higher the confidence value, the higher the
"reliability/precision" of the classification.

In other words, this new field provides an hint about "how" the flow
classification has been obtained.
For example, the application may want to ignore classification "by-port"
(they are not real DPI classifications, after all) or give a second
glance at flows classified via LRU caches (because of false positives).

Setting only one value for the confidence field is a bit tricky: more
work is probably needed in the next future to tweak/fix/improve the logic.
2022-01-11 15:23:39 +01:00
Ivan Nardi
b1e9245d94
ndpiReader: slight simplificaton of the output (#1378) 2021-11-27 17:32:23 +01:00
Ivan Nardi
b6d9536533
Fixed cleartext protocol assignment (#1357) 2021-10-25 15:04:04 +02:00
Luca Deri
9e97d20c25 Refreshed results list 2021-10-16 12:03:16 +02:00
Luca Deri
c2e7fbce94 Updated test results after latest commit 2021-10-16 11:57:32 +02:00
Luca Deri
e8455236bd Updated output 2021-08-07 17:38:33 +02:00
Luca Deri
61fc5be202 Reworked flow risk implementation 2021-07-23 17:27:15 +02:00
Ivan Nardi
cccf794265
ndpiReader: add statistics about nDPI performance (#1240)
The goal is to have a (roughly) idea about how many packets nDPI needs
to properly classify a flow.

Log this information (and guessed flows number too) during unit tests,
to keep track of improvements/regressions across commits.
2021-07-13 12:28:39 +02:00
Luca Deri
732bcecd17 Added flow risk score 2021-05-18 21:05:47 +02:00
Luca Deri
1a37595de0 Removed check for knowns protocols (major and app protocols) 2021-03-03 00:57:56 +01:00
Luca Deri
56bfb439f8 Improved DGA detection with trigrams. Disadvantage: slower startup time
Reworked Tor dissector embedded in TLS (fixes #1141)
Removed false positive on HTTP User-Agent
2021-03-03 00:41:07 +01:00
Ivan Nardi
b8a5358e80
QUIC: improve handling of SNI (#1105)
* QUIC: SNI should be always saved in flow->protos.stun_ssl.ssl.client_requested_server_name

Close #1077

* QUIC: fix matching of custom categories

* QUIC: add NDPI_TLS_MISSING_SNI support for older GQUIC versions

* QUIC: fix serialization

* QUIC: add DGA check for older GQUIC versions
2021-01-07 10:55:23 +01:00
Luca Deri
23a15bae5f Fixes #1029 2020-11-27 18:51:56 +01:00
Nardi Ivan
7d5a0e1f04 QUIC: extract User Agent information 2020-09-08 11:03:22 +02:00
Luca Deri
f597086386 Stddev calculation changes 2020-08-30 12:48:32 +02:00
Luca Deri
dd75060932 Fixed false positive in suspicous user agent
Optimized stddev calculation
2020-08-30 12:25:15 +02:00
Nardi Ivan
23ec82b59d Major rework of QUIC dissector
Improve support for GQUIC (up to Q046) and add support for Q050 and (IETF-)QUIC
Still no sub-classification for Q050 and QUIC
2020-08-21 22:04:55 +02:00
Luca Deri
e71df49b3e Changed due to bin size extension 2020-07-30 00:06:46 +02:00
Luca Deri
12abcd516b Updated test results due to bin changes 2020-07-09 17:28:02 +02:00
Luca Deri
1a62f4c799 Added ndpi_bin_XXX API
Added packet lenght distribution bins
2020-06-22 01:02:54 +02:00
emanuele-f
fd94270507 Remove decimals in test results for IAT, packet lengths and goodput ratio 2020-02-14 11:42:20 +01:00
Luca Deri
e98b994a39 Updated results 2019-11-21 13:35:04 +01:00
Luca Deri
4fd7e5734a Manual merge of pull #769 2019-10-02 23:01:29 +02:00
Luca
886d575157 Added -C to generate CSV analysis files
Improved IAT and byte distribution
2019-09-03 18:38:54 +02:00
Luca
c328452687 Average calculation fix 2019-08-29 16:08:18 +02:00
Luca Deri
8b8fd25626 Compilation fix on systems without JSON-C 2019-08-29 15:35:29 +02:00
Luca
b1270fc7bb Uodated results 2019-08-29 15:23:01 +02:00
Luca
e4e40e3c70 Added entropy, average, stddev, variance, bytes ratio calculation 2019-08-28 14:02:39 +02:00
Luca
c6e832be18 Various TLS/STUN improvememnts 2019-08-08 18:09:12 +02:00
Luca
419160f351 Implemented DTLS support
Renamed ssl to tls
2019-08-08 12:45:27 +02:00
Luca Deri
b8867642fc Refresh after data leak detection 2019-07-18 11:49:53 +02:00
Luca Deri
c43bec826d Updated results with the new SSL dissection
Added RDP dissector
2019-05-30 11:15:50 +02:00