Commit graph

3587 commits

Author SHA1 Message Date
Ivan Nardi
075bce5f3d
XIAOMI: add detection of Xiaomi traffic (#1529)
Most of the credits should go to @utoni (see #1521)
2022-04-25 11:00:02 +02:00
Toni
ac0f50b561
Added RakNet protocol dissector. (#1527)
* Frame Set PDU's do not get fully dissected for the sake of simplicity

Signed-off-by: lns <matzeton@googlemail.com>
2022-04-24 13:08:57 +02:00
Luca Deri
9d31b7450b Code cleanup (removed redundancy) 2022-04-22 19:03:50 +02:00
Toni
d5ae9170ba
Tiny gnutella improvement if gtk-gnutella used. (#1525)
Signed-off-by: lns <matzeton@googlemail.com>
2022-04-22 18:33:27 +02:00
Toni
824c343602
Updated `utils/whatsapp_ip_addresses_download.sh' to scrape the required IP addresses/ranges. (#1524)
* Replaced return statements in `utils/*.sh' with exit's (such scripts should never source'd)
 * Ran `utils/update_every_lists.sh'

Signed-off-by: lns <matzeton@googlemail.com>
2022-04-22 13:27:41 +02:00
Ivan Nardi
42909673ce
Add some scripts to easily update some IPs lists (#1522)
Follow-up of 8b062295

Add a new protocol id for generic Tencent/Wechat flows
2022-04-21 20:43:52 +02:00
Toni
ddc256b09e
Reduce ndpiReader's -h' spam. -H' does this job now. (#1523)
Signed-off-by: lns <matzeton@googlemail.com>
2022-04-21 20:43:36 +02:00
Toni
c1d46ebc4e
Added proprietary Agora Software Defined Real-time Network (SD-RTN) protocol dissector. (#1520)
Signed-off-by: lns <matzeton@googlemail.com>
2022-04-20 18:31:28 +02:00
Toni
dbf2fb8c9f
Added Toca Boca protocol dissector. (#1517)
Signed-off-by: lns <matzeton@googlemail.com>
2022-04-19 19:32:49 +02:00
Toni
a83ca555fb
Removed superfluous ifdef'd includes. (#1519)
Signed-off-by: lns <matzeton@googlemail.com>
2022-04-19 17:51:48 +02:00
Toni
fa79f07d15
Improved sflow protocol detection false-positives. (#1518)
Signed-off-by: lns <matzeton@googlemail.com>
2022-04-19 17:46:40 +02:00
Ivan Nardi
739dfc54b0
Kerberos: fix Undefined-shift error (#1516)
Similar to the error fixed in 4775be3d
Found by oss-fuzz.
See: https://bugs.chromium.org/p/oss-fuzz/issues/detail?id=46713
2022-04-18 20:54:13 +02:00
Luca Deri
8b2c9860be DGA improvements 2022-04-16 09:56:08 +02:00
Zied Aouini
9c8a3f270a
Minor fix. 2022-04-15 18:01:34 +02:00
Zied Aouini
fba75a3cf5
Merge pull request #1491 from utoni/fix/windows-msys2
Fixed msys2 build and re-activated CI Mingw-w64 build.
2022-04-15 13:22:25 +02:00
Toni Uhlig
c3df3a12aa Fixed msys2 build warnings and re-activated CI Mingw64 build.
* Removed Visual Studio leftovers. Maintaining an autotools project with VS integration requires some additional overhead.

Signed-off-by: Toni Uhlig <matzeton@googlemail.com>
Signed-off-by: lns <matzeton@googlemail.com>
2022-04-14 19:17:48 +02:00
Ivan Nardi
4775be3d85
Kerberos: fix some memory access errors (#1514)
```
==19724==ERROR: AddressSanitizer: heap-buffer-overflow on address 0x60e00000045e at pc 0x5620b8b3d3cc bp 0x7ffe0fda6b50 sp 0x7ffe0fda6310
READ of size 2 at 0x60e00000045e thread T0
    #0 0x5620b8b3d3cb in __interceptor_strncpy (/home/ivan/svnrepos/nDPI/fuzz/fuzz_process_packet_with_main+0x63f3cb) (BuildId: ee53ff920c8cd4c226d8520a0d4846d8864726b6)
    #1 0x5620b8d9b69c in strncpy_lower /home/ivan/svnrepos/nDPI/src/lib/protocols/kerberos.c:208:4
    #2 0x5620b8d995a0 in krb_parse /home/ivan/svnrepos/nDPI/src/lib/protocols/kerberos.c:316:5
    #3 0x5620b8d97a90 in ndpi_search_kerberos /home/ivan/svnrepos/nDPI/src/lib/protocols/kerberos.c:687:12
    #4 0x5620b8bcef35 in check_ndpi_detection_func /home/ivan/svnrepos/nDPI/src/lib/ndpi_main.c:4996:4
    #5 0x5620b8bd1be8 in check_ndpi_udp_flow_func /home/ivan/svnrepos/nDPI/src/lib/ndpi_main.c:5072:10
    #6 0x5620b8bd159c in ndpi_check_flow_func /home/ivan/svnrepos/nDPI/src/lib/ndpi_main.c:5105:12
    #7 0x5620b8be323a in ndpi_detection_process_packet /home/ivan/svnrepos/nDPI/src/lib/ndpi_main.c:5924:15
    #8 0x5620b8b8f7e0 in LLVMFuzzerTestOneInput /home/ivan/svnrepos/nDPI/fuzz/fuzz_process_packet.c:24:3
    #9 0x5620b8b8fd1b in main /home/ivan/svnrepos/nDPI/fuzz/fuzz_process_packet.c:84:17
    #10 0x7f45b32b90b2 in __libc_start_main /build/glibc-sMfBJT/glibc-2.31/csu/../csu/libc-start.c:308:16
    #11 0x5620b8acf47d in _start (/home/ivan/svnrepos/nDPI/fuzz/fuzz_process_packet_with_main+0x5d147d) (BuildId: ee53ff920c8cd4c226d8520a0d4846d8864726b6)

0x60e00000045e is located 0 bytes to the right of 158-byte region [0x60e0000003c0,0x60e00000045e)
allocated by thread T0 here:
    #0 0x5620b8b5283e in malloc (/home/ivan/svnrepos/nDPI/fuzz/fuzz_process_packet_with_main+0x65483e) (BuildId: ee53ff920c8cd4c226d8520a0d4846d8864726b6)
    #1 0x5620b8b8fc86 in main /home/ivan/svnrepos/nDPI/fuzz/fuzz_process_packet.c:70:17
    #2 0x7f45b32b90b2 in __libc_start_main /build/glibc-sMfBJT/glibc-2.31/csu/../csu/libc-start.c:308:16
```

```
protocols/kerberos.c:79:52: runtime error: left shift of 255 by 24 places cannot be represented in type 'int'
```

Found by oss-fuzz:
https://bugs.chromium.org/p/oss-fuzz/issues/detail?id=46670
https://bugs.chromium.org/p/oss-fuzz/issues/detail?id=46636
2022-04-13 18:10:56 +02:00
Luca Deri
e00997ea7c Extended list of cybersecurity domains 2022-04-13 18:05:29 +02:00
Wayne
939572c61e
fix(ndpi_main):Fix memory leak about ndpi_str; (#1513)
Co-authored-by: 林文烽 <wenfeng.lin@baishan.com>
2022-04-12 15:34:37 +02:00
Ivan Nardi
06a0abb6fd
TINC: fix invalid memory read (#1512)
```
=================================================================
==19324==ERROR: AddressSanitizer: heap-buffer-overflow on address 0x60600061be96 at pc 0x55b4a4cb4460 bp 0x7ffc7b461a70 sp 0x7ffc7b461a68
READ of size 1 at 0x60600061be96 thread T0
    #0 0x55b4a4cb445f in ndpi_check_tinc /home/ivan/svnrepos/nDPI/src/lib/protocols/tinc.c:105:9
    #1 0x55b4a4cb1888 in ndpi_search_tinc /home/ivan/svnrepos/nDPI/src/lib/protocols/tinc.c:135:5
    #2 0x55b4a4b4a6e1 in check_ndpi_detection_func /home/ivan/svnrepos/nDPI/src/lib/ndpi_main.c:5013:6
    #3 0x55b4a4b4c2d4 in check_ndpi_tcp_flow_func /home/ivan/svnrepos/nDPI/src/lib/ndpi_main.c:5084:12
    #4 0x55b4a4b4bf77 in ndpi_check_flow_func /home/ivan/svnrepos/nDPI/src/lib/ndpi_main.c:5103:12
    #5 0x55b4a4b5dcca in ndpi_detection_process_packet /home/ivan/svnrepos/nDPI/src/lib/ndpi_main.c:5924:15
    #6 0x55b4a4a87734 in packet_processing /home/ivan/svnrepos/nDPI/example/reader_util.c:1519:31
    #7 0x55b4a4a80761 in ndpi_workflow_process_packet /home/ivan/svnrepos/nDPI/example/reader_util.c:2093:10
    #8 0x55b4a4a39c8d in LLVMFuzzerTestOneInput /home/ivan/svnrepos/nDPI/fuzz/fuzz_ndpi_reader.c:107:7
    #9 0x55b4a4a3a46b in main /home/ivan/svnrepos/nDPI/fuzz/fuzz_ndpi_reader.c:179:17
    #10 0x7f69c63760b2 in __libc_start_main /build/glibc-sMfBJT/glibc-2.31/csu/../csu/libc-start.c:308:16
    #11 0x55b4a497954d in _start (/home/ivan/svnrepos/nDPI/fuzz/fuzz_ndpi_reader_with_main+0x61654d) (BuildId: 705ebc5c412d267294a65cb01f03a1f012aeaf20)

0x60600061be96 is located 0 bytes to the right of 54-byte region [0x60600061be60,0x60600061be96)
allocated by thread T0 here:
[...]
```
Found by oss-fuzz:
https://bugs.chromium.org/p/oss-fuzz/issues/detail?id=46499
2022-04-10 16:59:34 +02:00
Toni
e3faecf9de
Improved ASN.1 parsing for Keberos. Fixes #1492. (#1497)
* This is a quick fix, the Kerberos protocol dissector requires some refactoring effort.

Signed-off-by: Toni Uhlig <matzeton@googlemail.com>
Signed-off-by: lns <matzeton@googlemail.com>
2022-04-10 10:36:10 +02:00
Vinicius Silva Nogueira
713cdd0b13
QUIC: handle retransmissions and overlapping fragments in reassembler (#1195) (#1498)
* QUIC: handle retransmissions and overlapping fragments in reassembler

* Trigger CI

* minor fix: parentheses

* Changing ndpi_malloc to ndpi_calloc

* fix memory leak

* quic_reasm_buf calloc to malloc

* change order of is_ch_complete && is_reasm_buf_complete call

* is_reasm_buf_complete: added handling for case where frame size is not multiple of 8

* add extra check
2022-04-07 21:18:58 +02:00
aouinizied
9537788ade Fix JSON-C. 2022-04-07 17:59:11 +02:00
aouinizied
ff993e2f57 Python bindings fix. 2022-04-07 17:54:41 +02:00
Luca Deri
a6e2f4a15a Added ndpi_find_outliers() API call using Z-Score 2022-04-04 10:02:45 +02:00
Luca Deri
19019383f0 Added -z flag 2022-04-03 11:45:23 +02:00
Ivan Nardi
e0e231756c
ndpiReader: fix compilation (#1510)
Not sure why Windows started complaining... anyway, the fixes has been
taken from https://github.com/ntop/nDPI/pull/1491: credits to @lnslbrty
2022-04-01 22:01:39 +02:00
Luca Deri
8080b3d14f Removed un-necessary guess in mining 2022-04-01 19:23:13 +02:00
Luca Deri
9f3a6f131b update 2022-04-01 19:07:46 +02:00
Luca Deri
32a7d42469 Fixed incompatibilities due to https://github.com/ntop/nDPI/pull/1509 2022-04-01 19:07:00 +02:00
Luca Deri
6eb5bd88a7 DGA improvements 2022-04-01 17:39:29 +02:00
Luca Deri
a75d77b536 Waring fixes 2022-04-01 17:35:25 +02:00
Vitaliy Ivanov
61bc9815d5
ndpireader: add json output back. (#1509)
- partial revert of:
    commit 51cfdfb0d8
    Author: Luca Deri <deri@ntop.org>
    Date: Sun Nov 17 17:51:45 2019 +0100
      Removed unused JSON-C code

- Json option is changed from 'j' to 'k' as it's used in the new codebase.
- use HAVE_LIBJSON_C instead of HAVE_JSON_C.
- tabs vs spaces clean ups.

Signed-off-by: Vitaliy Ivanov <vitaliyi@interfacemasters.com>

 Conflicts:
	example/ndpiReader.c
2022-04-01 11:47:01 +02:00
Luca Deri
8238e68275 Improvements for CUSTOM_NDPI_PROTOCOLS 2022-04-01 09:02:05 +02:00
Luca Deri
1e8cdd893c Moved geneated file to a separate folder 2022-04-01 08:58:53 +02:00
Luca Deri
0ff3e77c42 Improved twitter detection 2022-04-01 08:50:56 +02:00
Luca Deri
6677d692c4 Removed SRV record from suspicious DNS traffic 2022-03-31 11:32:30 +02:00
Luca Deri
98a548c642 Improved DGA detection 2022-03-31 11:29:30 +02:00
Darryl Sokoloski
46dc8b5bda
[autoconf] Fixed .git submodule detection test. (#1507)
Signed-off-by: Darryl Sokoloski <darryl@sokoloski.ca>
2022-03-31 03:21:51 +02:00
Luca Deri
4e199abd39 Added code for identifiying anomalies with metrics stored in InfluxDB 2022-03-30 22:12:39 +02:00
Ivan Nardi
6bcba118d4
reader_util: add support for userAgent in SSDP (#1502)
Update unit tests results
Follow-up of d668ab4b
2022-03-28 04:25:34 +02:00
Ivan Nardi
6acf7a6abe
Add support for Pluralsight site (#1503) 2022-03-27 15:13:12 +02:00
Ivan Nardi
ec80f1fb3b
Fix CI tests results (#1504)
CI integration is failing since 856d7d2.
2022-03-27 15:12:39 +02:00
Vitaly Lavrov
4896dabb6b
Reducing the size of the ndpi_detection_module_struct structure. (#1490)
The ndpi_detection_module_struct structure contains 5 arrays "struct
ndpi_call_function_struct" size 286*144=41 kB size,
which are occupied by a small number of elements.

At the moment we have callback_buffer_size = 172, tcp_with_payload=114,
tcp_no_payload=8, udp=93, other 8.
NDPI_MAX_SUPPORTED_PROTOCOLS = 285.

Size of struct ndpi_detection_module_struct is 253136 bytes.
Size of all structs ndpi_call_function_struct 5*286*144=205920 bytes.
Real use memory size for struct ndpi_call_function_struct is
(173+224)*144=57168 bytes.
2022-03-27 12:00:21 +02:00
Darryl Sokoloski
d668ab4bec
[SSDP] Extract HTTP user-agent when available. (#1500)
[SSDP] Added capture file with UA header.
[SSDP] Added pcap test output log file.

Signed-off-by: Darryl Sokoloski <darryl@sokoloski.ca>
2022-03-27 11:10:44 +02:00
Luca Deri
856d7d2916 Improved DGA detection skipping names containign at least 3 consecutive digits in the first word 2022-03-26 09:59:55 +01:00
Ivan Nardi
aeb5f1f947
QUIC: add support for version 2 draft 01 (#1493)
Support for v2-00 has been removed (it has never been used in real
networks and it is incompatible with v2-01).

Chrome already supports v2-01 in latest versions in Chrome Beta channel.
2022-03-25 10:16:30 +01:00
Ivan Nardi
af1d20bca1
Mining: cleanup registration (#1496)
Use the same pattern of all the other dissectors: one registration and
one callback.
Spotted by @dsokoloski
2022-03-25 10:07:47 +01:00
Vinicius Silva Nogueira
26df1403e6
Trying to improve QUIC reassembler (#1195) (#1489)
* handling QUIC out-of-order fragments

* minor fix

* updated quic_frags_ch_out_of_order_same_packet_craziness.pcapng.out

* quic test: buf_len + last_pos

* QUIC: comment update in __reassemble function and minor change is_ch_complete function
2022-03-24 10:51:31 +01:00
aouinizied
bb12837ca7 Update Python bindings guide. 2022-03-22 15:01:55 +01:00