Commit graph

94 commits

Author SHA1 Message Date
Adrian Pekar
68dfdb6120
Upgrade nDPI from 4.7.0 to 5.0 with Windows libpcap fix (#235)
* Upgrade nDPI from 4.7.0 to 5.0 with Windows build fix (#230)


- Upgrade nDPI submodule from 4.7.0 to 5.0
- Fix all 9 nDPI 5.0 API breaking changes:
  * Protocol structure: .app_protocol -> .proto.app_protocol
  * Protocol structure: .master_protocol -> .proto.master_protocol
  * ndpi_protocol2name() signature change
  * ndpi_detection_giveup() signature change (removed enable_guess/guessed params)
  * ndpi_extra_dissection_possible() replaced with state check
  * ndpi_init_detection_module() now expects NULL
  * Protocol bitmask removed (all protocols enabled by default)
  * TCP/UDP struct size checks removed
  * Fingerprint extraction: ja3_client -> ja4_client

- Maintain backward compatibility by keeping field names:
  * client_fingerprint: Now JA4 (TLS), HASSH client (SSH), DHCP fingerprint
  * server_fingerprint: Now JA3S (TLS), HASSH server (SSH)

- Update build system for nDPI 5.0:
  * Add --with-only-libndpi flag to build.sh
  * Make CFFI marker extraction optional in engine_build.py

- Update tests for nDPI 5.0:
  * Update fingerprint assertions (JA3 MD5 -> JA4 format)
  * Regenerate all 228 baseline test results
  * Remove invalid test file (memcached.cap)

- All 29 tests passing

* Fix memcpy buffer overreads and enable DNS subclassification

- Fix buffer overread in fingerprint memcpy calls by using source size
  instead of destination size:
  - HASSH client/server: 33 bytes (was reading 48)
  - JA4 client: 37 bytes (was reading 48)
  - JA3 server: 33 bytes (was reading 48)
  - DHCP: 48 bytes (consistent with others)

- Enable DNS subclassification (disabled by default in nDPI 5.0)
  to restore detection of DNS.Apple, DNS.Google, etc.

- Regenerate test baselines with fixes applied

* Replace outdated Steam test pcaps with nDPI 5.0 version

- Remove steam.pcap and steam_datagram_relay_ping.pcapng (obsolete)
- Add steam.pcapng from nDPI 5.0 test suite
- nDPI 5.0 reworked Steam detection (ntop/nDPI#2264)

* Fix Windows build: add missing configure step for nDPI 5.0

The Windows build script was missing the ./configure --with-only-libndpi
step before make, causing nDPI headers to not be properly installed.
This resulted in the CFFI preprocessing marker //CFFI.NDPI_PACKED_STRUCTURES
being absent from ndpi_cdefinitions_packed.h, crashing engine_build.py.

Aligns build_windows.sh with build.sh which already had this step.

* Fix Windows build: install libpcap-dev for nDPI 5.0 configure

nDPI 5.0 configure requires libpcap when --with-only-libndpi is used
on MinGW (see ntop/nDPI#3114). Install mingw-w64-x86_64-libpcap via
pacman to satisfy this dependency.
2026-02-15 20:30:16 +01:00
Zied Aouini
3cb5f6cac8
Revert "Upgrade nDPI from 4.7.0 to 5.0 with Windows build fix (#230)" (#234)
This reverts commit b1b2ee87e2.
2026-02-14 21:34:30 +01:00
Adrian Pekar
b1b2ee87e2
Upgrade nDPI from 4.7.0 to 5.0 with Windows build fix (#230)
* Upgrade nDPI from 4.7.0 to 5.0 with backward compatibility

- Upgrade nDPI submodule from 4.7.0 to 5.0
- Fix all 9 nDPI 5.0 API breaking changes:
  * Protocol structure: .app_protocol -> .proto.app_protocol
  * Protocol structure: .master_protocol -> .proto.master_protocol
  * ndpi_protocol2name() signature change
  * ndpi_detection_giveup() signature change (removed enable_guess/guessed params)
  * ndpi_extra_dissection_possible() replaced with state check
  * ndpi_init_detection_module() now expects NULL
  * Protocol bitmask removed (all protocols enabled by default)
  * TCP/UDP struct size checks removed
  * Fingerprint extraction: ja3_client -> ja4_client

- Maintain backward compatibility by keeping field names:
  * client_fingerprint: Now JA4 (TLS), HASSH client (SSH), DHCP fingerprint
  * server_fingerprint: Now JA3S (TLS), HASSH server (SSH)

- Update build system for nDPI 5.0:
  * Add --with-only-libndpi flag to build.sh
  * Make CFFI marker extraction optional in engine_build.py

- Update tests for nDPI 5.0:
  * Update fingerprint assertions (JA3 MD5 -> JA4 format)
  * Regenerate all 228 baseline test results
  * Remove invalid test file (memcached.cap)

- All 29 tests passing

* Fix memcpy buffer overreads and enable DNS subclassification

- Fix buffer overread in fingerprint memcpy calls by using source size
  instead of destination size:
  - HASSH client/server: 33 bytes (was reading 48)
  - JA4 client: 37 bytes (was reading 48)
  - JA3 server: 33 bytes (was reading 48)
  - DHCP: 48 bytes (consistent with others)

- Enable DNS subclassification (disabled by default in nDPI 5.0)
  to restore detection of DNS.Apple, DNS.Google, etc.

- Regenerate test baselines with fixes applied

* Replace outdated Steam test pcaps with nDPI 5.0 version

- Remove steam.pcap and steam_datagram_relay_ping.pcapng (obsolete)
- Add steam.pcapng from nDPI 5.0 test suite
- nDPI 5.0 reworked Steam detection (ntop/nDPI#2264)

* Fix Windows build: add missing configure step for nDPI 5.0

The Windows build script was missing the ./configure --with-only-libndpi
step before make, causing nDPI headers to not be properly installed.
This resulted in the CFFI preprocessing marker //CFFI.NDPI_PACKED_STRUCTURES
being absent from ndpi_cdefinitions_packed.h, crashing engine_build.py.

Aligns build_windows.sh with build.sh which already had this step.
2026-02-14 17:48:00 +01:00
Zied Aouini
8a13708698
Revert "Upgrade nDPI from 4.7.0 to 5.0 (#220)" (#228)
This reverts commit 22cd213550.
2026-02-13 22:16:13 +01:00
Adrian Pekar
22cd213550
Upgrade nDPI from 4.7.0 to 5.0 (#220)
* Upgrade nDPI from 4.7.0 to 5.0 with backward compatibility

- Upgrade nDPI submodule from 4.7.0 to 5.0
- Fix all 9 nDPI 5.0 API breaking changes:
  * Protocol structure: .app_protocol -> .proto.app_protocol
  * Protocol structure: .master_protocol -> .proto.master_protocol
  * ndpi_protocol2name() signature change
  * ndpi_detection_giveup() signature change (removed enable_guess/guessed params)
  * ndpi_extra_dissection_possible() replaced with state check
  * ndpi_init_detection_module() now expects NULL
  * Protocol bitmask removed (all protocols enabled by default)
  * TCP/UDP struct size checks removed
  * Fingerprint extraction: ja3_client -> ja4_client

- Maintain backward compatibility by keeping field names:
  * client_fingerprint: Now JA4 (TLS), HASSH client (SSH), DHCP fingerprint
  * server_fingerprint: Now JA3S (TLS), HASSH server (SSH)

- Update build system for nDPI 5.0:
  * Add --with-only-libndpi flag to build.sh
  * Make CFFI marker extraction optional in engine_build.py

- Update tests for nDPI 5.0:
  * Update fingerprint assertions (JA3 MD5 -> JA4 format)
  * Regenerate all 228 baseline test results
  * Remove invalid test file (memcached.cap)

- All 29 tests passing

* Fix memcpy buffer overreads and enable DNS subclassification

- Fix buffer overread in fingerprint memcpy calls by using source size
  instead of destination size:
  - HASSH client/server: 33 bytes (was reading 48)
  - JA4 client: 37 bytes (was reading 48)
  - JA3 server: 33 bytes (was reading 48)
  - DHCP: 48 bytes (consistent with others)

- Enable DNS subclassification (disabled by default in nDPI 5.0)
  to restore detection of DNS.Apple, DNS.Google, etc.

- Regenerate test baselines with fixes applied

* Replace outdated Steam test pcaps with nDPI 5.0 version

- Remove steam.pcap and steam_datagram_relay_ping.pcapng (obsolete)
- Add steam.pcapng from nDPI 5.0 test suite
- nDPI 5.0 reworked Steam detection (ntop/nDPI#2264)
2026-02-13 22:03:42 +01:00
Zied Aouini
8a749befae nDPI update. 2023-01-27 14:37:46 +01:00
Zied Aouini
57f6afd321 Update nDPI. 2022-10-25 18:12:50 +02:00
Zied Aouini
2e8d685640 Update nDPI. 2022-09-27 17:06:48 +02:00
Zied Aouini
88c74826ca Fix nDPI integration. 2022-09-12 16:21:01 +02:00
Zied Aouini
fc94e134d9 Fix test cases. 2022-07-28 13:57:38 +02:00
Zied Aouini
cf8145bd36 Fix consecutive pcap files reading (https://github.com/nfstream/nfstream/pull/131) 2022-07-28 12:50:21 +02:00
Zied Aouini
db7518823e Sync nDPI. 2022-07-22 18:31:58 +02:00
aouinizied
9d2756242b [WIP] Move to CFFI API mode. 2022-04-05 18:04:01 +02:00
aouinizied
c8009c61a8 [WIP] Improve CI. 2022-03-25 17:25:21 +01:00
aouinizied
00449108bd Fix raw datalink handling on Windows. (https://github.com/nfstream/nfstream/issues/104 and https://github.com/nfstream/nfstream/issues/114) 2022-03-15 18:54:55 +01:00
aouinizied
35e323726b Fix Cis. 2022-03-11 14:38:54 +01:00
aouinizied
78a499f11e FiX arm64 CI. 2022-03-08 21:13:19 +01:00
aouinizied
ff51419f6e Drop application_is_guessed flow feature. 2022-03-08 20:08:06 +01:00
aouinizied
7495d94428 Fix windows CI. 2022-03-07 18:21:50 +01:00
aouinizied
7dc6fa8b31 Add test traces. 2022-03-07 16:19:30 +01:00
aouinizied
b07918f581 nDPI sync. 2022-03-07 15:20:12 +01:00
Zied Aouini
5fa605d7a9 nDPI sync. 2022-02-21 15:02:24 +01:00
Zied Aouini
7b602fe658 nDPI sync. 2022-02-10 11:33:22 +01:00
Zied Aouini
f90193c0e7 nDPI sync. 2022-01-13 15:18:01 +01:00
Zied Aouini
57dd6c9580 nDPI sync. 2022-01-12 19:04:01 +01:00
Zied Aouini
10078e341a Update test results. 2021-12-22 13:56:05 +01:00
Zied Aouini
09b12f34cc Update tests. 2021-12-06 11:47:33 +01:00
Zied Aouini
f65c6bce43 Introduce Windows Support. 2021-12-03 17:46:31 +01:00
Zied Aouini
c9216c44b2 [WIP] Windows support. 2021-11-30 14:29:26 +01:00
Zied Aouini
e2a90bafc3 [WIP] Windows support. 2021-11-29 16:21:56 +01:00
Zied Aouini
46e0120347 [WIP] Windows support. 2021-11-29 14:53:15 +01:00
Zied Aouini
dba1b1f9e4 [WIP] Windows support. 2021-11-29 14:30:39 +01:00
Zied Aouini
940a26d8af Update tests files. 2021-11-03 19:34:18 +01:00
aouinizied
7d7e4deb56 Sync nDPI (ipv6 + reduce memory). 2021-09-21 15:40:01 +02:00
aouinizied
2cfed4d144 Fix truncated test files. 2021-09-01 18:43:01 +02:00
aouinizied
7ce7de1412 Upgrade macos CI to macos-11. 2021-09-01 18:26:08 +02:00
aouinizied
7ba12bc293 Fix macOS CI (debug). 2021-09-01 17:38:33 +02:00
aouinizied
6c256c1ba1 Fix macOS CI. 2021-09-01 17:15:46 +02:00
aouinizied
e1ad676dfc Update nDPI definitions. 2021-06-29 15:01:44 +02:00
Zied Aouini
1b2ddd6747 Impelment tunnel_id extraction. 2021-04-21 16:29:59 +02:00
aouinizied
6edb552f21 Add support for pcapng format (https://github.com/nfstream/nfstream/issues/54) 2021-02-19 15:18:39 +01:00
aouinizied
f973d3ca2e Add test files (quic, mongodb, amongus, skype). 2020-11-10 16:33:16 +01:00
aouinizied
2a5254909b Add VPN test file. 2020-11-02 20:06:34 +01:00
aouinizied
32bdff7fb2 Fix CI. 2020-11-02 17:17:54 +01:00
aouinizied
5495e46d89 Fix nDPI integration tests. 2020-10-22 20:15:48 +02:00
aouinizied
0e6ea9f802 Minor improvements. 2020-10-22 01:50:28 +02:00
aouinizied
979799847a Minor fixes. 2020-10-15 20:37:50 +02:00
aouinizied
c2fd2b3b17 Simplify testing process. 2020-10-13 21:21:16 +02:00
Zied Aouini
a5f48370ae Fix tests for DHCP Plugin. 2020-10-09 17:36:48 +02:00
aouinizied
4bfd95d9ac Unify CI workflow. 2020-09-11 01:20:42 +02:00