Commit graph

657 commits

Author SHA1 Message Date
Luca Deri
6a1fd9ad97 Added missing check to prevent crashes 2021-06-23 12:17:21 +02:00
Vitaly Lavrov
2234b97149
ndpiReader: memory leak (#1215)
Non-critical bugs.
If a file list is used, then all files except the last are not closed.
Opening the next file loses the memory allocated via pcap_open_offline() for the previous file.
If a bpf filter is used, then no memory is freed after pcap_compile.
2021-06-23 12:04:03 +02:00
Alfredo Cardigliano
4aefbe0c7a Call ac_automata_release with free_pattern = 1 (malloc'ed patterns expected in ndpi_add_string_to_automa) 2021-06-14 14:41:14 +02:00
Luca Deri
380286c069 Fixes https://github.com/ntop/ntopng/issues/5482 2021-06-11 22:21:03 +02:00
Ivan Nardi
9d427faafe
ndpiReader: fix collecting of risks statistics (#1192) 2021-06-01 16:50:46 +02:00
Luca
c620858671 Reworked ndpi flow risk score adding client and server score 2021-06-01 09:17:26 +02:00
Luca Deri
732bcecd17 Added flow risk score 2021-05-18 21:05:47 +02:00
Luca Deri
86f3c29d03 Typo 2021-05-18 19:52:33 +02:00
Luca Deri
ca15e3295e Added risk/score dump (ndpiReader -h)
Added ndpi_dump_risks_score() API score
2021-05-18 19:34:17 +02:00
Luca Deri
43a8576efb Reworked human readeable string search in flows
Removed fragment manager code
2021-05-17 20:55:06 +02:00
Luca Deri
ac1eaca8a6 Added browser TLS heuristic 2021-05-13 20:00:27 +02:00
Luca Deri
a62be9b8ec Implemented heuristic to detect Safari and Firefox TLS browsing 2021-05-13 12:37:07 +02:00
Toni
87076dcd5b
Fixed obsolete error printing if CTRL-C is pressed. #1165 (#1184)
* This fix was proposed by @robertsong2019

Signed-off-by: Toni Uhlig <matzeton@googlemail.com>
2021-05-11 21:38:56 +02:00
Luca Deri
4297a65ce8 Implemented flow score in Wireshark integration 2021-05-10 22:43:05 +02:00
Toni
8c28613eb2
Check datalink during fuzzing to prevent console / logfile spam. See #1175 for more information. (#1177)
Signed-off-by: Toni Uhlig <matzeton@googlemail.com>
2021-05-09 15:09:43 +02:00
Luca
ae2470fad4 Initial work towards detection via TLS of browser types 2021-05-06 21:42:06 +02:00
Luca Deri
dd65142020 Compilation fix 2021-04-27 08:26:08 +02:00
Luca Deri
70686249c9 Updated code due to https://github.com/ntop/nDPI/pull/1175 2021-04-27 08:12:14 +02:00
Toni
da3e6bd61b
Check for common ALPNs and set a flow risk if not known. (#1175)
* Increased risk bitmask to 64bit (instead of 32bit).
 * Removed annoying "Unknown datalink" error message for fuzzers.

Signed-off-by: Toni Uhlig <matzeton@googlemail.com>
2021-04-27 07:22:04 +02:00
Luca Deri
87ad2b58dc Compilation fix 2021-04-26 10:50:19 +02:00
Luca Deri
4a09707e48 Added flow risk to wireshark dissection 2021-04-26 10:17:29 +02:00
Ivan Nardi
9ca62ed7ac
Fix detunneling of GTP-U traffic (#1168)
Fuzzing #1161 exposed some (completely unrelated) issues on GTP-U
detunneling code.
(see https://github.com/ntop/nDPI/actions/runs/719882047)
2021-04-18 21:37:51 +02:00
Ivan Nardi
fb74785282
Fix some warnings about unused variables/functions (#1160) 2021-04-05 19:21:30 +02:00
Luca Deri
a1dba74346 Trace fix 2021-04-02 12:55:15 +02:00
Luca Deri
4f8ca9485a Fixed incapoatibilities with the latest extcap/wireshark 2021-04-01 23:53:53 +02:00
Luca Deri
fcbc16da00 Fixed invalid guess stats 2021-03-30 17:49:48 +02:00
Ivan Nardi
a6029d250d
ndpiReader: print an error msg if we found an unsupported datalink type (#1157) 2021-03-23 11:47:29 +01:00
Toni
b040407683
Refactored nDPI subprotocol handling and aimini protocol detection. (#1156)
* Refactored and merged callback buffer routines for non-udp-tcp / udp / tcp / tcp-wo-payload.

Signed-off-by: Toni Uhlig <matzeton@googlemail.com>

* Try to detect one subprotocol if a detected protocol can have one.

 * This adds a performance overhead due to much more protocol detection routine calls.
   See #1148 for more information.

Signed-off-by: Toni Uhlig <matzeton@googlemail.com>

* Refactor subprotocol handling (1/2).

Signed-off-by: Toni Uhlig <matzeton@googlemail.com>

* Refactor subprotocol handling (2/2).

Signed-off-by: Toni Uhlig <matzeton@googlemail.com>

* Prevent some code duplication by using macros for ndpi_int_one_line_struct string comparision.

Signed-off-by: Toni Uhlig <matzeton@googlemail.com>

* Refactored aimini HTTP detection parts (somehow related to #1148).

Signed-off-by: Toni Uhlig <matzeton@googlemail.com>

* Added aimini client/server test pcap.

Signed-off-by: Toni Uhlig <matzeton@googlemail.com>

* Removed master protocol as it was only used for STUN and via also removed API function ndpi_get_protocol_id_master_proto

 * Adjusted Python code to conform to the changes made during the refactoring process.

Signed-off-by: Toni Uhlig <matzeton@googlemail.com>
2021-03-23 11:46:12 +01:00
Luca Deri
627299e4dd Better DGA detection (slightly decreased accuracy) 2021-03-20 17:56:24 +01:00
Luca Deri
b22cb70a68 Added % of flows with risks 2021-03-14 21:59:33 +01:00
Luca Deri
1b6275005e Added in stats the number of flows with risks 2021-03-14 21:54:35 +01:00
Luca Deri
3b02cfab69 Added flows risks report 2021-03-14 21:46:23 +01:00
Luca Deri
565a7bfce3 Reworked extendal dependency across testing tools 2021-03-14 20:48:21 +01:00
Luca Deri
e58527536d Help crash fix 2021-03-14 19:55:21 +01:00
Luca Deri
9419015711 Implemented square erro rollup to avoid overflow 2021-03-14 11:01:51 +01:00
Luca
192fad4402 Added double exponential smoothing implementation 2021-03-11 09:39:52 +01:00
Luca Deri
6833ee2bbe Added single exponential smoothing API
int ndpi_ses_init(struct ndpi_ses_struct *ses, double alpha, float significance);
int ndpi_ses_add_value(struct ndpi_ses_struct *ses, const u_int32_t _value, double *forecast, double *confidence_band);
2021-03-11 00:04:33 +01:00
Luca Deri
f6ad16d8f8 Added experiemntal JA3+ implementation that can be used with -z i ndpiReader 2021-03-09 23:38:29 +01:00
Ivan Nardi
c50a8d4808
Add support for Snapchat voip calls (#1147)
* Add support for Snapchat voip calls

Snapchat multiplexes some of its audio/video real time traffic with QUIC
sessions. The peculiarity of these sessions is that they are Q046 and
don't have any SNI.

* Fix tests with libgcrypt disabled
2021-03-06 05:48:36 +01:00
Luca Deri
0f8a994841 Improved DGA detection
Before
Accuracy 66%, Precision 86%, Recall 38%

After
Accuracy 71%, Precision 89%, Recall 49%
2021-03-03 19:30:01 +01:00
Luca Deri
56bfb439f8 Improved DGA detection with trigrams. Disadvantage: slower startup time
Reworked Tor dissector embedded in TLS (fixes #1141)
Removed false positive on HTTP User-Agent
2021-03-03 00:41:07 +01:00
Luca Deri
4bff595733 Holt-Winters calculation improvement 2021-02-27 11:32:51 +01:00
Toni
16890a6632
Added NDPI_MALICIOUS_SHA1 flow risk. (#1142)
* An external file which contains known malicious SSL certificate SHA-1 hashes
   can be loaded via ndpi_load_malicious_sha1_file(...)

Signed-off-by: Toni Uhlig <matzeton@googlemail.com>
2021-02-26 17:00:05 +01:00
Luca Deri
f1b22b199f Added NDPI_MALICIOUS_JA3 flow risk
Added ndpi_load_malicious_ja3_file() API call
2021-02-22 23:19:23 +01:00
Luca Deri
fc3db8f169 Implemented TLS Certificate Sibject matching
Improved AnyDesk detection
2021-02-22 22:37:33 +01:00
Luca Deri
fc16c9368e Added risky domain flow-risk support 2021-02-21 21:45:46 +01:00
Ivan Nardi
421609475e
Fix small memory leak (#1133)
Now function definition matches the prototype in ndpi_api.h.in
2021-02-10 15:24:34 +01:00
Toni
1e12c90c66
Fixed memory leaks caused by conditional free'ing for some TLS connec… (#1132)
* Fixed memory leaks caused by conditional free'ing for some TLS connections.

 * Members of tls_quic struct should also free'd if the detected master protocol is IMAPS / POPS / SMTPS / etc.

Signed-off-by: Toni Uhlig <matzeton@googlemail.com>

* Prevent reader_util.c from exit()'ing if maximum flow count reached.
This confuses the fuzzer.

 * Improved fuzz/Makefile.am to use LDADD for ../example/libndpiReader.a instead of LDFLAGS.
   That way, fuzz_ndpi_reader re-links to ../example/libndpiReader.a if something changed there.

Signed-off-by: Toni Uhlig <matzeton@googlemail.com>
2021-02-10 15:24:11 +01:00
Luca Deri
e2f6569adb Fixed CPHA missing protocol initialization
Improved IEC104 and IRC detection
2021-02-10 15:22:20 +01:00
Luca Deri
1331e0aec9 Extended the API to calculate jitter
- ndpi_jitter_init()
- ndpi_jitter_free()
- ndpi_jitter_add_value()
2021-02-09 15:56:03 +01:00