Commit graph

50 commits

Author SHA1 Message Date
Ivan Nardi
ca5ffc4988
TLS: improve handling of ALPN(s) (#1784)
Tell "Advertised" ALPN list from "Negotiated" ALPN; the former is
extracted from the CH, the latter from the SH.

Add some entries to the known ALPN list.

Fix printing of "TLS Supported Versions" field.
2022-10-25 17:06:29 +02:00
Ivan Nardi
a7c2734b38
Remove classification "by-ip" from protocol stack (#1743)
Basically:
* "classification by-ip" (i.e. `flow->guessed_protocol_id_by_ip` is
NEVER returned in the protocol stack (i.e.
`flow->detected_protocol_stack[]`);
* if the application is interested into such information, it can access
`ndpi_protocol->protocol_by_ip` itself.

There are mainly 4 points in the code that set the "classification
by-ip" in the protocol stack:  the generic `ndpi_set_detected_protocol()`/
`ndpi_detection_giveup()` functions and the HTTP/STUN  dissectors.

In the unit tests output, a print about `ndpi_protocol->protocol_by_ip`
has been added for each flow: the huge diff of this commit is mainly due
to that.

Strictly speaking, this change is NOT an API/ABI breakage, but there are
important differences in the classification results. For examples:
* TLS flows without the initial handshake (or without a matching
SNI/certificate) are simply classified as `TLS`;
* similar for HTTP or QUIC flows;
* DNS flows without a matching request domain are simply classified as
`DNS`; we don't have `DNS/Google` anymore just because the server is
8.8.8.8 (that was an outrageous behaviour...);
* flows previusoly classified only "by-ip" are now classified as
`NDPI_PROTOCOL_UNKNOWN`.

See #1425 for other examples of why adding the "classification by-ip" in
the protocol stack is a bad idea.

Please, note that IPV6 is not supported :(  (long standing issue in nDPI) i.e.
`ndpi_protocol->protocol_by_ip` wil be always `NDPI_PROTOCOL_UNKNOWN` for
IPv6 flows.

Define `NDPI_CONFIDENCE_MATCH_BY_IP` has been removed.

Close #1687
2022-09-20 22:24:47 +02:00
Ivan Nardi
4f584f78a0
Fix ndpi_do_guess() (#1731)
Avoid a double call of `ndpi_guess_host_protocol_id()`.
Some code paths work for ipv4/6 both
Remove some never used code.
2022-09-12 19:28:41 +02:00
Ivan Nardi
405a52ed65
Patricia tree, Ahocarasick automa, LRU cache: add statistics (#1683)
Add (basic) internal stats to the main data structures used by the
library; they might be usefull to check how effective these structures
are.

Add an option to `ndpiReader` to dump them; enabled by default in the
unit tests.
This new option enables/disables dumping of "num dissectors calls"
values, too (see b4cb14ec).
2022-07-29 15:25:00 +02:00
Ivan Nardi
172e698bb8
TINC: avoid processing SYN packets (#1676)
Since e6b332aa, we have proper support for detecting client/server
direction. So Tinc dissector is now able to properly initialize the
cache entry only when needed and not anymore at the SYN time; initializing
that entry for **every** SYN packets was a complete waste of resources.

Since 4896dabb, the various `struct ndpi_call_function_struct`
structures are not more separate objects and therefore comparing them
using only their pointers is bogus: this bug was triggered by this
change because `ndpi_str->callback_buffer_size_tcp_no_payload` is now 0.
2022-07-28 12:39:18 +02:00
Ivan Nardi
d8d525fff2
Update the protocol bitmask for some protocols (#1675)
Tcp retransmissions should be ignored.

Remove some unused protocol bitmasks.

Update script to download Whatsapp IP list.
2022-07-27 11:46:45 +02:00
Toni
ae2bedce3a
Improved Jabber/XMPP detection. (#1661)
Signed-off-by: Toni Uhlig <matzeton@googlemail.com>
2022-07-13 17:55:33 +02:00
Ivan Nardi
b4cb14ec19
Keep track of how many dissectors calls we made for each flow (#1657) 2022-07-11 09:47:47 +02:00
Ivan Nardi
ff4e010501
Avoid spurious calls to extra dissection (#1648)
If the extra callabck is not set, calling the extra dissection is only a
waste of resources...
2022-07-07 17:49:35 +02:00
Ivan Nardi
3a087e951d
Add a "confidence" field about the reliability of the classification. (#1395)
As a general rule, the higher the confidence value, the higher the
"reliability/precision" of the classification.

In other words, this new field provides an hint about "how" the flow
classification has been obtained.
For example, the application may want to ignore classification "by-port"
(they are not real DPI classifications, after all) or give a second
glance at flows classified via LRU caches (because of false positives).

Setting only one value for the confidence field is a bit tricky: more
work is probably needed in the next future to tweak/fix/improve the logic.
2022-01-11 15:23:39 +01:00
Ivan Nardi
b1e9245d94
ndpiReader: slight simplificaton of the output (#1378) 2021-11-27 17:32:23 +01:00
Luca Deri
e8455236bd Updated output 2021-08-07 17:38:33 +02:00
Ivan Nardi
cccf794265
ndpiReader: add statistics about nDPI performance (#1240)
The goal is to have a (roughly) idea about how many packets nDPI needs
to properly classify a flow.

Log this information (and guessed flows number too) during unit tests,
to keep track of improvements/regressions across commits.
2021-07-13 12:28:39 +02:00
Luca Deri
43a8576efb Reworked human readeable string search in flows
Removed fragment manager code
2021-05-17 20:55:06 +02:00
Luca Deri
ac1eaca8a6 Added browser TLS heuristic 2021-05-13 20:00:27 +02:00
Luca Deri
044ed14b4f Various optimizations to reduce not-necessary calls
Optimized various UDP dissectors
Removed dead protocols such as pando and pplive
2020-09-24 23:26:03 +02:00
Luca Deri
2ae4c6675d Fixed partial TLS dissection 2020-07-30 18:30:07 +02:00
Luca Deri
e71df49b3e Changed due to bin size extension 2020-07-30 00:06:46 +02:00
Luca Deri
12abcd516b Updated test results due to bin changes 2020-07-09 17:28:02 +02:00
Luca Deri
1a62f4c799 Added ndpi_bin_XXX API
Added packet lenght distribution bins
2020-06-22 01:02:54 +02:00
Luca Deri
4a09b4efa0 Added TLS issuerDN and subjectDN 2020-05-07 18:44:51 +02:00
Luca Deri
748df7972a Results update 2020-04-17 10:36:27 +02:00
emanuele-f
fd94270507 Remove decimals in test results for IAT, packet lengths and goodput ratio 2020-02-14 11:42:20 +01:00
Luca Deri
5cad39f0e8 Added export of TLS supported version in TLS header 2020-02-07 23:26:03 +01:00
Luca Deri
3be263aafc Added TLS ALPN support 2020-02-07 21:54:04 +01:00
Luca
daae1cc9b1 Reworked TLS dissection 2020-01-01 12:59:19 +01:00
Luca Deri
e98b994a39 Updated results 2019-11-21 13:35:04 +01:00
Luca Deri
4fd7e5734a Manual merge of pull #769 2019-10-02 23:01:29 +02:00
Luca Deri
11e9fd8cef As TLS certificate fingerprint is computed, TLS without certificate protocol has been removed
Various improvemenets in detection quality
2019-09-15 12:18:24 +02:00
Luca Deri
0f9107bef3 Fixed timezone conversion 2019-09-10 22:15:23 +02:00
Luca Deri
068611e8ba Changes in validity format 2019-09-10 21:44:42 +02:00
Luca Deri
65bdd0de9a Uddated results 2019-09-10 00:43:03 +02:00
Luca Deri
086c511a11 Parsed TLS certificate validity 2019-09-10 00:34:48 +02:00
Luca
886d575157 Added -C to generate CSV analysis files
Improved IAT and byte distribution
2019-09-03 18:38:54 +02:00
Luca
b1270fc7bb Uodated results 2019-08-29 15:23:01 +02:00
Luca
e4e40e3c70 Added entropy, average, stddev, variance, bytes ratio calculation 2019-08-28 14:02:39 +02:00
Luca
419160f351 Implemented DTLS support
Renamed ssl to tls
2019-08-08 12:45:27 +02:00
Luca Deri
3c4a959c1e Updated results when adding client hello 2019-08-07 20:06:31 +02:00
Luca Deri
5c758d4564 Reworked output
-v 1 now prints only unknown flows
-v 2 now prints all flows
2019-07-15 14:45:25 +02:00
Luca Deri
17c49b2e6d Updated test resultss after export changes 2019-07-13 18:37:57 +02:00
Luca Deri
c43bec826d Updated results with the new SSL dissection
Added RDP dissector
2019-05-30 11:15:50 +02:00
Luca
1290706fad Tests result fix
Merge branch 'dev' of https://github.com/ntop/nDPI into dev
2019-04-05 12:51:59 +02:00
Luca
a499f369a5 Updated results based on the new output format 2018-07-21 15:20:11 +02:00
Luca Deri
36c1b72118 Updated test resuls 2018-05-18 23:22:14 +02:00
Luca Deri
529492e81f Updated test results 2018-05-14 21:42:19 +02:00
Luca
3ba3a08141 Implemented flow sort based on total bytes so that we can (with -v X) immediately spot elephants and mice 2017-08-02 20:15:21 +02:00
Micah Lyle
2e13e44b9c Added testing scripts and commited files that only had the server diff on the right 2017-07-14 18:19:49 -07:00
Campus
4829222e73 update results after modifications 2017-05-26 11:26:22 +02:00
Luca Deri
1d2e67d38c Dissected SSH client/server versions and reported in data structures 2017-02-11 10:48:08 +01:00
emanuele-f
ba3bf6c3d9 Fix facebook certificate recognition 2016-09-06 00:44:17 +02:00