Commit graph

5656 commits

Author SHA1 Message Date
Ivan Nardi
d6cbd624d0
TLS: fix JA4 when there are no ciphers or extensions (#3084) 2026-01-13 19:33:23 +01:00
Ivan Nardi
6828c1ef30
TLS: fix JA4 when there are more than 99 ciphers or extensions (#3083) 2026-01-13 19:31:00 +01:00
Ivan Nardi
7173a48b2b
SMTP: set extra dissection only if we did set the protocol classification (#3080)
Found by oss-fuzz
See: https://issues.oss-fuzz.com/issues/474480392?pli=1
2026-01-13 19:30:40 +01:00
Ivan Nardi
411af3e639
Fix HTTP hostname normalization with IPv6 literal address (#3081)
Close #3065
2026-01-13 18:43:02 +01:00
Ivan Nardi
68603ba321
CI: arm docker: move to ubuntu 20.04 (from 18.04) (#3079)
Are newer versions available?
2026-01-13 17:10:46 +01:00
Ivan Nardi
575a2b238a
Fix fuzzer dependencies to rebuild when library changes (#3078)
Five fuzzers (fuzz_process_packet, fuzz_ndpi_reader, fuzz_ndpi_reader_alloc_fail,
fuzz_ndpi_reader_payload_analyzer, and fuzz_tls_certificate) were not rebuilding
when libndpi.a changed because their explicit DEPENDENCIES declarations only
included dictionary files.

In Automake, when prog_DEPENDENCIES is explicitly set, it overrides the automatic
dependency generation from LDADD. This caused these fuzzers to miss the library
dependency that the other 55 fuzzers correctly inherited.

This commit adds $(top_builddir)/src/lib/libndpi.a to the DEPENDENCIES for all
5 affected fuzzers, ensuring they rebuild whenever the library changes.

🤖 Generated with [Claude Code](https://claude.com/claude-code)

Co-authored-by: Claude Sonnet 4.5 <noreply@anthropic.com>
2026-01-13 17:10:05 +01:00
Ivan Nardi
9e9fe394c2
Build system: add --with-libpcap configure option for Linux custom libpcap paths (#3077)
Users can now specify a custom libpcap installation path on Linux using
--with-libpcap=PATH, enabling testing with different libpcap versions
without system-wide installation. The implementation prefers static
libraries, auto-detects dependencies via pkg-config, and displays the
selected libpcap path and libraries in the configuration summary.

🤖 Generated with [Claude Code](https://claude.com/claude-code)

Co-authored-by: Claude Sonnet 4.5 <noreply@anthropic.com>
2026-01-13 17:09:43 +01:00
Ivan Nardi
2e6166b85b
Fix python bindings (#3076) 2026-01-13 17:09:14 +01:00
Luca Deri
62fbb610f0 Added ndpi_data_burstiness() for computing data burstiness 2026-01-11 22:07:16 +01:00
Luca Deri
922187a5a2 (C) update 2026-01-09 15:52:40 +01:00
Luca Deri
04ee0bb48b nDPI fingerprint calculation
- Added the ability to exclude TCP fingerprint via metadata.ndpi_fingerprint_ignore_tcp_fp configuration
- TLS blocks not include the block lenght only for Client and Server hello, ignored for all other blocks
2026-01-09 12:00:10 +01:00
Luca Deri
b085899ede ndpi_serialize_tls_blocks() is now part of the nDPi API 2026-01-09 09:44:20 +01:00
Luca Deri
d3206230bb TLS blocks timing is not hidden by default.
Use --cfg "tls,tls_blocks_show_timing,1" to show it
2026-01-09 09:24:35 +01:00
Luca Deri
901e317422
Added --cfg "tls,max_num_blocks_to_analyze,X" for dynamically setting TLS blocks number (#3073)
* Added --cfg "tls,max_num_blocks_to_analyze,X" where if X > 0 TLS blocks are analyzed

Example --cfg "tls,max_num_blocks_to_analyze,8"

* TLS blocks now include a time-delta (msec) with respect to the previous TLS block.
The format is @<msec delta>. Example:

 "tls_blocks": [
         "22:1=232@191",
         "22:2=-122@5,20=-1@5,21=-23@5,21=-905@5,21=-281@5",
         "21=-53@0",
         "20=1@3,21=53@3",
         "21=-218@119,21=-218@119",
 ]
2026-01-08 23:36:13 +01:00
Luca Deri
6597a83f27 Added metadata.ndpifp_ignore_sni_tls_extn to ignore SNI from nDPI fingerprint.
Example: ndpiReader --cfg "tls,metadata.ndpifp_ignore_sni_tls_extn,1"  ...

Fixed tls blocks lenght calculation
2026-01-08 19:25:10 +01:00
Luca
8e7229c0a1 Index fix 2026-01-07 11:04:53 +01:00
Luca Deri
35132077ae Fixed TLS JSON serialization 2026-01-05 17:48:31 +01:00
Luca Deri
412c63df19
Enhanced TLS blocks computation and included in nDPI Fingerprint (#3071)
* Added TLS blocks serialization

   "tls_blocks": [
           "22:1=496",
           "22:2=-122,20=-1,21=-27,21=-871,21=-281,21=-53",
           "20=1"
         ]

Howto read it
- TLS records belonging to the same (reassembled) packet go on the same line
- The format is <record type>=<record len>
- The record lenght is positive is sent cli->srv, negative otherwise
- In order to avoid the SNI lenght (present in ClientHello) to influence the lenght, the ClientHello record lenght does not include the SNI lenght (if SNI is present)

* TLS blocks are now reported in numerical form
Extended TLS blocks analysis to blocks othr thank client/server hello
nDPI fingerprint now includes initial TLS blocks
Added padding (RFC 7685) in the list of TLS ephemeral extensions
2026-01-04 23:15:08 +01:00
Luca Deri
6eb2256ce6 Added JA4 testing pcap 2026-01-02 15:36:24 +01:00
Luca Deri
c9efbaa016 Added new tls configuration option named metadata.ja_ignore_ephemeral_tls_extn
that enables in JA$ to ignore ephemeral TLS extensions (i.e. those extensions
that are used to make TLS  more efficient across sessions but that create
different JAx fingerprints for the same application.

For compatibility reason with the JA4, this feature is disbled by default
but you can enabled it (e.g. in ndpiReader) using
--cfg "tls,metadata.ja_ignore_ephemeral_tls_extn,1"
2026-01-01 11:37:26 +01:00
Luca Deri
1ffc582e76 Case fix 2026-01-01 11:16:30 +01:00
Luca Deri
d2a91c5b41 Typo 2026-01-01 11:16:19 +01:00
Luca Deri
37ca034697 (C) update 2026-01-01 10:31:40 +01:00
Luca Deri
8aaff75ede
Fixes protocol inconsistencies (#3069)
* Added code to address inconsistencies
* Added check for discarding dig errors
* Temporarily disabled whatsapp script: an update is required. See https://developers.facebook.com/docs/whatsapp/on-premises/sunset
2025-12-31 20:54:35 +01:00
Luca Deri
bfbb6eb2df Added detection of plugins not enabled 2025-12-30 18:47:40 +01:00
Luca
7d00f37528 Removed unncessary serialization 2025-12-29 18:48:33 +01:00
Luca Deri
612c1d2264 tls_blocks in JSON are now symbolic 2025-12-27 21:04:59 +01:00
Luca Deri
8b7e588e42
Enhanced TLS Blocks Computation (#3068) 2025-12-27 20:43:59 +01:00
Luca Deri
e49fa91627 Added tls_blocks serialization in JSON/csv
Use --cfg "tls,blocks_analysis,1" with ndpiReader
2025-12-26 21:06:19 +01:00
Luca Deri
5a0df66a45 Exported bins in JSON/csv 2025-12-26 19:53:03 +01:00
Luca Deri
459c6ac9b7 Added check for unset value that prevents a crash as the dns dissector manipulates protos.dns without setting the protocol 2025-12-24 09:12:33 +01:00
Luca Deri
159c05f032
Added ability to export SSH key exchanges (disabled by default).
It's possible to enable it using "--cfg=ssh,metadata.ssh_data,1" in ndpiReader. When enabled
the negotiated SSH key exchange method is returned.
2025-12-20 20:19:17 +01:00
Luca Deri
45e3213d7f
Reworked TLS dissector to export further metadata (#3066)
* Reworked TLS dissector to give users the ability to )optionnaly)
collect and export TLS metadata via "--cfg=tls,metadata.ja_data,1"
2025-12-20 17:33:09 +01:00
Luca Deri
712f4bc72a
Reverted Win change (#3064) 2025-12-18 06:53:20 +01:00
Luca Deri
56d7a268ed Win fixes 2025-12-15 12:26:34 +01:00
Toni
246462592e
Add additional msgpack protocol validations (Fix #3060, false-positives) (#3061)
Signed-off-by: Toni Uhlig <matzeton@googlemail.com>
2025-12-11 14:18:00 +01:00
Ivan Nardi
ce04aea085 condif: fix "only_classification" configuration 2025-12-09 15:15:11 +01:00
Ivan Nardi
87440c59bb
fuzz: extend fuzzing coverage and fix loading of TCP fingerprints from file (#3059) 2025-12-09 14:03:46 +01:00
Luca Deri
0a97483835 Defined type for extra packet function 2025-12-08 21:24:52 +01:00
Ivan Nardi
85a8ed8a10 Skip all-local and clean-local targets when building with --with-only-libndpi
The all-local and clean-local targets copy configuration files from example/,
lists/, and tests/cfgs/ directories for out-of-tree builds. These files are only
needed for example programs and tests, not for library-only builds.

When configured with --with-only-libndpi, EXTRA_TARGETS is empty, so we can skip
these setup/cleanup operations by checking if EXTRA_TARGETS is non-empty.

🤖 Generated with [Claude Code](https://claude.com/claude-code)

Co-Authored-By: Claude Sonnet 4.5 <noreply@anthropic.com>
2025-12-08 17:51:32 +01:00
Ivan Nardi
99b0668765 Fix 2025-12-08 17:51:32 +01:00
Ivan Nardi
f3b6cb95b4 Fix tarball 2025-12-08 17:51:32 +01:00
Ivan Nardi
39f3cd9558 test: rework main script
The issue about `config.txt` files is that they contains paths:
* to configuration files, which are in the source tree
* to the dynamic plugins, which are in the build tree

Solution:
* copy all configuration files into the build tree
* all those paths are about the build tree
* tests run from the build tree, no from the source tree anymore
2025-12-08 17:51:32 +01:00
Toni Uhlig
285496d0b9 Add (generic) MsgPack protocol dissector.
Signed-off-by: Toni Uhlig <matzeton@googlemail.com>
2025-12-08 17:50:20 +01:00
Ivan Nardi
155484a140 CI: test --disable-plugin-support and add a job on FreeBSD 2025-12-08 16:55:40 +01:00
Toni
19ab035178
Force disabling plugin system --disable-plugin-support (#3057)
* third party software might not need plugins or does not want it
 * no forced linkage against `dl`

Signed-off-by: Toni Uhlig <matzeton@googlemail.com>
2025-12-08 14:36:08 +01:00
Toni
5ec473be4d
Substitution pattern ${string/pattern/replace} does not work with FreeBSD 14/15 (#3055)
Signed-off-by: Toni Uhlig <matzeton@googlemail.com>
2025-12-08 14:33:43 +01:00
Ivan Nardi
11be5b31c0
Fix fuzzing instrumentation broken by e49e93cc (#3054)
Commit e49e93cc17 broke coverage
instrumentation for fuzzing targets due to two issues with how
AM_LDFLAGS and target-specific CFLAGS/CXXFLAGS interact.

Problems:
---------
1. Missing AM_LDFLAGS in link command:
   The commit changed LIB_FUZZING_ENGINE from being added to LDFLAGS
   to being added to AM_LDFLAGS (line 26):
     Before: `LDFLAGS += $(LIB_FUZZING_ENGINE)`
     After:  `AM_LDFLAGS += $(LIB_FUZZING_ENGINE)`

   However, FUZZ_LINK_COMMAND (line 34) was not updated to include
   `$(AM_LDFLAGS)`, so `-fsanitize=fuzzer` was missing from link commands.

2. Target-specific CFLAGS/CXXFLAGS override AM_CFLAGS/AM_CXXFLAGS:
   When automake sees target-specific CFLAGS (like fuzz_ndpi_reader_CFLAGS),
   it COMPLETELY REPLACES AM_CFLAGS instead of adding to it. Even empty
   assignments like `fuzz_process_packet_CFLAGS =` mean "use nothing"
   rather than "use AM_CFLAGS". This means `-fsanitize=fuzzer` from
   AM_CFLAGS was not being used during compilation.

   Example:
     `AM_CFLAGS = @NDPI_CFLAGS@ -fsanitize=fuzzer`
     `fuzz_ndpi_reader_CFLAGS = -I$(top_srcdir)/example/`

   Result: Only `-I$(top_srcdir)/example/` is used, AM_CFLAGS is ignored!

Without `-fsanitize=fuzzer` during both compilation and linking:
- No coverage instrumentation is generated
- LibFuzzer cannot collect coverage information
- Fuzzer warns: "WARNING: no interesting inputs were found so far.
  Is the code instrumented for coverage?"

Solutions:
----------
1. Add `$(AM_LDFLAGS)` to FUZZ_LINK_COMMAND (line 34) before
   `$(LDFLAGS)`
   This ensures LIB_FUZZING_ENGINE is included during linking.

2. For targets with non-empty CFLAGS/CXXFLAGS, prefix with `$(AM_CFLAGS)/$(AM_CXXFLAGS)`:
   Changed: `fuzz_*_CFLAGS = -DFOO`
   To:      `fuzz_*_CFLAGS = $(AM_CFLAGS) -DFOO`

3. For targets with empty CFLAGS/CXXFLAGS, remove the assignments entirely:
   Removed: `fuzz_*_CFLAGS =`

   This allows automake to automatically use AM_CFLAGS/AM_CXXFLAGS.

The flag ordering (package flags before user flags) is maintained.

Testing:
--------
Before fix:
  $ ./fuzz_ndpi_reader -runs=10
  INFO: Seed: 437565050
  WARNING: no interesting inputs were found so far. Is the code instrumented for coverage?

After fix:
  $ ./fuzz_ndpi_reader -runs=10
  INFO: Loaded 1 modules   (4802 inline 8-bit counters)
  INFO: Loaded 1 PC tables (4802 PCs)
  #2  INITED cov: 4 ft: 5 corp: 1/1b exec/s: 0 rss: 81Mb
  #10 DONE   cov: 4 ft: 5 corp: 1/1b lim: 4 exec/s: 0 rss: 81Mb

  $ ./fuzz_process_packet -runs=10
  INFO: Loaded 1 modules   (25 inline 8-bit counters)
  INFO: Loaded 1 PC tables (25 PCs)
  #2  INITED cov: 2 ft: 2 corp: 1/1b exec/s: 0 rss: 65Mb
  #10 DONE   cov: 2 ft: 2 corp: 1/1b lim: 4 exec/s: 0 rss: 65Mb

Verified with:
  CC=clang CXX=clang++ ./configure --enable-fuzztargets --with-sanitizer
  make -j4
  ./fuzz/fuzz_ndpi_reader -runs=10
  ./fuzz/fuzz_process_packet -runs=10

🤖 Generated with [Claude Code](https://claude.com/claude-code)

Co-authored-by: Claude Sonnet 4.5 <noreply@anthropic.com>
2025-12-07 22:31:59 +01:00
Toni
aa3241e17b
Add (generic) JSON protocol dissector. (#2492)
Signed-off-by: Toni Uhlig <matzeton@googlemail.com>
2025-12-06 20:15:19 +01:00
Luca Deri
11cc612e3a Added ability to redefine existing dissectors inside plugins 2025-12-05 15:47:21 +01:00