Five fuzzers (fuzz_process_packet, fuzz_ndpi_reader, fuzz_ndpi_reader_alloc_fail,
fuzz_ndpi_reader_payload_analyzer, and fuzz_tls_certificate) were not rebuilding
when libndpi.a changed because their explicit DEPENDENCIES declarations only
included dictionary files.
In Automake, when prog_DEPENDENCIES is explicitly set, it overrides the automatic
dependency generation from LDADD. This caused these fuzzers to miss the library
dependency that the other 55 fuzzers correctly inherited.
This commit adds $(top_builddir)/src/lib/libndpi.a to the DEPENDENCIES for all
5 affected fuzzers, ensuring they rebuild whenever the library changes.
🤖 Generated with [Claude Code](https://claude.com/claude-code)
Co-authored-by: Claude Sonnet 4.5 <noreply@anthropic.com>
- Added the ability to exclude TCP fingerprint via metadata.ndpi_fingerprint_ignore_tcp_fp configuration
- TLS blocks not include the block lenght only for Client and Server hello, ignored for all other blocks
* Added --cfg "tls,max_num_blocks_to_analyze,X" where if X > 0 TLS blocks are analyzed
Example --cfg "tls,max_num_blocks_to_analyze,8"
* TLS blocks now include a time-delta (msec) with respect to the previous TLS block.
The format is @<msec delta>. Example:
"tls_blocks": [
"22:1=232@191",
"22:2=-122@5,20=-1@5,21=-23@5,21=-905@5,21=-281@5",
"21=-53@0",
"20=1@3,21=53@3",
"21=-218@119,21=-218@119",
]
Commit e49e93cc17 broke coverage
instrumentation for fuzzing targets due to two issues with how
AM_LDFLAGS and target-specific CFLAGS/CXXFLAGS interact.
Problems:
---------
1. Missing AM_LDFLAGS in link command:
The commit changed LIB_FUZZING_ENGINE from being added to LDFLAGS
to being added to AM_LDFLAGS (line 26):
Before: `LDFLAGS += $(LIB_FUZZING_ENGINE)`
After: `AM_LDFLAGS += $(LIB_FUZZING_ENGINE)`
However, FUZZ_LINK_COMMAND (line 34) was not updated to include
`$(AM_LDFLAGS)`, so `-fsanitize=fuzzer` was missing from link commands.
2. Target-specific CFLAGS/CXXFLAGS override AM_CFLAGS/AM_CXXFLAGS:
When automake sees target-specific CFLAGS (like fuzz_ndpi_reader_CFLAGS),
it COMPLETELY REPLACES AM_CFLAGS instead of adding to it. Even empty
assignments like `fuzz_process_packet_CFLAGS =` mean "use nothing"
rather than "use AM_CFLAGS". This means `-fsanitize=fuzzer` from
AM_CFLAGS was not being used during compilation.
Example:
`AM_CFLAGS = @NDPI_CFLAGS@ -fsanitize=fuzzer`
`fuzz_ndpi_reader_CFLAGS = -I$(top_srcdir)/example/`
Result: Only `-I$(top_srcdir)/example/` is used, AM_CFLAGS is ignored!
Without `-fsanitize=fuzzer` during both compilation and linking:
- No coverage instrumentation is generated
- LibFuzzer cannot collect coverage information
- Fuzzer warns: "WARNING: no interesting inputs were found so far.
Is the code instrumented for coverage?"
Solutions:
----------
1. Add `$(AM_LDFLAGS)` to FUZZ_LINK_COMMAND (line 34) before
`$(LDFLAGS)`
This ensures LIB_FUZZING_ENGINE is included during linking.
2. For targets with non-empty CFLAGS/CXXFLAGS, prefix with `$(AM_CFLAGS)/$(AM_CXXFLAGS)`:
Changed: `fuzz_*_CFLAGS = -DFOO`
To: `fuzz_*_CFLAGS = $(AM_CFLAGS) -DFOO`
3. For targets with empty CFLAGS/CXXFLAGS, remove the assignments entirely:
Removed: `fuzz_*_CFLAGS =`
This allows automake to automatically use AM_CFLAGS/AM_CXXFLAGS.
The flag ordering (package flags before user flags) is maintained.
Testing:
--------
Before fix:
$ ./fuzz_ndpi_reader -runs=10
INFO: Seed: 437565050
WARNING: no interesting inputs were found so far. Is the code instrumented for coverage?
After fix:
$ ./fuzz_ndpi_reader -runs=10
INFO: Loaded 1 modules (4802 inline 8-bit counters)
INFO: Loaded 1 PC tables (4802 PCs)
#2 INITED cov: 4 ft: 5 corp: 1/1b exec/s: 0 rss: 81Mb
#10 DONE cov: 4 ft: 5 corp: 1/1b lim: 4 exec/s: 0 rss: 81Mb
$ ./fuzz_process_packet -runs=10
INFO: Loaded 1 modules (25 inline 8-bit counters)
INFO: Loaded 1 PC tables (25 PCs)
#2 INITED cov: 2 ft: 2 corp: 1/1b exec/s: 0 rss: 65Mb
#10 DONE cov: 2 ft: 2 corp: 1/1b lim: 4 exec/s: 0 rss: 65Mb
Verified with:
CC=clang CXX=clang++ ./configure --enable-fuzztargets --with-sanitizer
make -j4
./fuzz/fuzz_ndpi_reader -runs=10
./fuzz/fuzz_process_packet -runs=10
🤖 Generated with [Claude Code](https://claude.com/claude-code)
Co-authored-by: Claude Sonnet 4.5 <noreply@anthropic.com>
Replace relative path references (../) with $(top_srcdir) in
fuzz/Makefile.am to properly support out-of-tree builds (VPATH builds).
🤖 Generated with [Claude Code](https://claude.com/claude-code)
Co-authored-by: Claude <noreply@anthropic.com>
Fix improper handling of CFLAGS and LDFLAGS throughout the build system.
Also remove hardcoded debug flags that prevented production builds
without symbols.
Problems:
---------
1. CFLAGS/LDFLAGS handling:
The build system was using `CFLAGS +=` and `LDFLAGS +=` to append
package-specific flags, which modifies the user's environment variables
instead of keeping package and user flags separate. This caused:
- User-specified optimization levels being overridden by package defaults
- Inability to properly override flags at configure or make time
- Problems with cross-compilation and embedded toolchains
2. Hardcoded -g flags:
Debug symbols (-g) were hardcoded in several Makefiles, forcing debug
symbols in all builds including production. This caused:
- Larger binary sizes (library and tools)
- No way to build without debug symbols
- Conflicts with user's debug level preferences (-g1, -g2, -g3)
- Redundancy with configure options (--enable-debug-build)
Solutions:
----------
1. Implement proper CFLAGS/LDFLAGS separation using AM_CFLAGS/AM_LDFLAGS:
- Added `CFLAGS = @CFLAGS@` to preserve configure-time flags
- Added `LDFLAGS = @LDFLAGS@` to preserve configure-time flags
- Changed `CFLAGS +=` to `AM_CFLAGS =` and `AM_CFLAGS +=`
- Changed `LDFLAGS +=` to `AM_LDFLAGS =` and `AM_LDFLAGS +=`
- Updated compilation rules: $(CC) $(AM_CFLAGS) $(CFLAGS) ...
- Updated linking rules: $(CC) ... $(AM_LDFLAGS) $(LDFLAGS) ...
2. Remove all hardcoded -g flags from Makefiles:
- Debug symbols now controlled via configure (--enable-debug-build)
or user CFLAGS (e.g., CFLAGS="-g3")
Flag ordering ensures:
- Package flags come first (e.g., -O2, -fPIC)
- User flags come after and can override (e.g., -O3)
- Last flag wins for conflicting options
🤖 Generated with [Claude Code](https://claude.com/claude-code)
Co-authored-by: Claude <noreply@anthropic.com>
This commit implements comprehensive improvements to the nDPI build system
to enhance portability, enable parallel testing, and ensure reliable
out-of-tree (VPATH) builds across all platforms.
Changes:
1. Optimize library linking order (configure.ac, all Makefiles)
- Reorder ADDITIONAL_LIBS to follow proper dependency hierarchy
- Move low-level libraries (libm) to end of link line
- Ensures compatibility with --as-needed linker flag
- Improves LTO and static linking support
2. Fix VPATH build dependencies (all Makefiles)
- Add explicit dependencies on generated headers (ndpi_config.h, ndpi_define.h)
- Prevents race conditions in parallel builds (make -j)
- Ensures headers exist before compilation starts
3. Replace mkdir -p with portable $(MKDIR_P) macro
4. Enable parallel test execution (configure.ac)
- Add 'parallel-tests' option to AM_INIT_AUTOMAKE
- Allows test suites to run concurrently during 'make check'
5. Add defensive .NOTPARALLEL directive (Makefile.am)
- Prevents race conditions if 'make -j clean distclean' is run
6. Fix clean target completeness (src/lib/Makefile.in)
- Remove all .so symlinks (libndpi.so, libndpi.so.N)
- Add cleanup for Windows DLL files (*.dll)
- Explicitly remove versioned shared libraries
🤖 Generated with [Claude Code](https://claude.com/claude-code)
Co-Authored-By: Claude <noreply@anthropic.com>
Co-Authored-By: Ivan Nardi <nardi.ivan@gmail.com>
- Fixed some typos and inconsistent option names in error messages
- Improved Git Detection in configure script
- Added informative warnings when optional dependencies are missing
- Improve error handling on autogen.sh script
- Simplify fuzzing Makefile, creating common FUZZ_LINK_COMMAND template for all fuzz targets
- Added *_DEPENDENCIES declarations for fuzz targets with dictionaries
- Implemented incremental corpus building to avoid unnecessary rebuilds
🤖 Generated with [Claude Code](https://claude.com/claude-code)
Co-authored-by: Claude <noreply@anthropic.com>
Full accounting of memory used by the library.
Change `ndpi_realloc()` prototype to be compatible with standard
`realloc()`.
Be compatible with croaring allocation logic.
Note that aligned allocations are used only by croaring code.
Note that flow allocations are used only by the application, not by the
library.
API changes:
* remove `set_ndpi_malloc()` and `set_ndpi_free()`; use
`ndpi_set_memory_alloction_functions()` instead
We should pay attention to tell ndpiReader configuration files and
libnDPI configuration files!! Better solution?
Be sure that configuration files are located where they are expected.
In oss-fuzz enviroment we can't make any assumptions about the current
working directory of your fuzz target.
Application should keep calling nDPI until flow state became
`NDPI_STATE_CLASSIFIED`.
The main loop in the application is simplified to something like:
```
res = ndpi_detection_process_packet(...);
if(res->state == NDPI_STATE_CLASSIFIED) {
/* Done: you can get finale classification and all metadata.
nDPI doesn't need more packets for this flow */
} else {
/* nDPI needs more packets for this flow. The provided
classification is not final and more metadata might be
extracted.
If `res->state` is `NDPI_STATE_PARTIAL`, partial/initial
classification is available in `res->proto`
as usual but it can be updated later.
*/
}
/*
Example A (QUIC flow):
pkt 1: proto QUIC state NDPI_STATE_PARTIAL
pkt 2: proto QUIC/Youtube state NDPI_STATE_CLASSIFIED
Example B (GoogleMeet call):
pkt 1: proto STUN state NDPI_STATE_PARTIAL
pkt N: proto DTLS state NDPI_STATE_PARTIAL
pkt N+M: proto DTLS/GoogleCall state NDPI_STATE_CLASSIFIED
Example C (standard TLS flow):
pkt 1: proto Unknown state NDPI_STATE_INSPECTING
pkt 2: proto Unknown state NDPI_STATE_INSPECTING
pkt 3: proto Unknown state NDPI_STATE_INSPECTING
pkt 4: proto TLS/Facebook state NDPI_STATE_PARTIAL
pkt N: proto TLS/Facebook state NDPI_STATE_CLASSIFIED
*/
}
```
You can take a look at `ndpiReader` for a slightly more complex example.
API changes:
* remove the third parameter from `ndpi_detection_giveup()`. If you need
to know if the classification flow has been guessed, you can access
`flow->protocol_was_guessed`
* remove `ndpi_extra_dissection_possible()`
* change some prototypes from accepting `ndpi_protocol foo` to
`ndpi_master_app_protocol bar`. The update is trivial: from `foo` to
`foo.proto`
Initial work to support out-of-tree builds
```
./autogen.sh
mkdir build
cd build
../configure
make
make check
```
IMPORTANT: `autogen.sh` doesn't call `configure` automatically anymore!!
You have to do: `./autogen.sh && ./configure --$OPTIONS`.
A little bit annoying but the pattern `autogen && configure && make` is
very common on Linux.
Known issues:
* `make doc` doesn't work in out-of-tree builds, yet
* Windows/MinGW/DPDK (out-of-tree) builds have not been tested, so it is unlikely they work
See: #2992
See: f2bccee04
This is clearly a workaround for a introspector bug/limittaions. It
seems that we need separate files for every fuzzers to get per-fuzzer
coverage stats
The idea: one c file for each fuzzer.
If it works, we can extend the same logic to every `fuzz_ndpoi_reader*`
fuzzers, otherwise we can revert that in a few days...
Right now, there is, in essence, a static mapping between flow protocols
and flow breeds.
Make it dynamic: allow to have different flows, with the same
classification but differents breeds. This is the same logic that we
already have for categories....
Preliminary work to support breed in category lists.
API change from the app POV: to get the flow breed don't use anymore
`ndpi_get_proto_breed()`, but access directly `struct ndpi_proto->breed`
The functions `ndpi_domain_classify_*()` and
`ndpi_get_host_domain_suffix()` now have a `u_int32_t` parameter as
`class_id` (instead of `u_int_16_t`), with the following logic:
```
class_id = (breed << 16) | category
```
instead of the old:
```
class_id = category
```
Please note that this change is back-compatible: if you are not
interested into breeds, you don't need to update the application code.
The idea is to remove the limitation of only two protocols ("master" and
"app") in the flow classifcation.
This is quite handy expecially for STUN flows and, in general, for any
flows where there is some kind of transitionf from a cleartext protocol
to TLS: HTTP_PROXY -> TLS/Youtube; SMTP -> SMTPS (via STARTTLS msg).
In the vast majority of the cases, the protocol stack is simply
Master/Application.
Examples of real stacks (from the unit tests) different from the standard
"master/app":
* "STUN.WhatsAppCall.SRTP": a WA call
* "STUN.DTLS.GoogleCall": a Meet call
* "Telegram.STUN.DTLS.TelegramVoip": a Telegram call
* "SMTP.SMTPS.Google": a SMTP connection to Google server started in
cleartext and updated to TLS
* "HTTP.Google.ntop": a HTTP connection to a Google domain (match via
"Host" header) and to a ntop server (match via "Server" header)
The logic to create the stack is still a bit coarse: we have a decade of
code try to push everything in only ywo protocols... Therefore, the
content of the stack is still **highly experimental** and might change
in the next future; do you have any suggestions?
It is quite likely that the legacy fields "master_protocol" and
"app_protocol" will be there for a long time.
Add some helper to use the stack:
```
ndpi_stack_get_upper_proto();
ndpi_stack_get_lower_proto();
bool ndpi_stack_contains(struct ndpi_proto_stack *s, u_int16_t proto_id);
bool ndpi_stack_is_tls_like(struct ndpi_proto_stack *s);
bool ndpi_stack_is_http_like(struct ndpi_proto_stack *s);
```
Be sure new stack logic is compatible with legacy code:
```
assert(ndpi_stack_get_upper_proto(&flow->detected_protocol.protocol_stack) ==
ndpi_get_upper_proto(flow->detected_protocol));
assert(ndpi_stack_get_lower_proto(&flow->detected_protocol.protocol_stack) ==
ndpi_get_lower_proto(flow->detected_protocol));
```
Change the API to enable/disable protocols: you can set that via the
standard `ndpi_set_config()` function, as every configuration
parameters. By default, all protocols are enabled.
Split the (local) context initialization into two phases:
* `ndpi_init_detection_module()`: generic part. It does not depend on the
configuration and on the protocols being enabled or not. It also
calculates the real number of internal protocols
* `ndpi_finalize_initialization()`: apply the configuration. All the
initialization stuff that depend on protocols being enabled or not
must be put here
This is the last step to have the protocols number fully calculated at
runtime
Remove a (now) useless fuzzer.
Important API changes:
* remove `NDPI_LAST_IMPLEMENTED_PROTOCOL` define
* remove `ndpi_get_num_internal_protocols()`. To get the number of
configured protocols (internal and custom) you must use
`ndpi_get_num_protocols()` after having called `ndpi_finalize_initialization()`
The main difference is that the memory is allocated at runtime
Typical usercase:
```
struct ndpi_bitmask b;
ndpi_bitmask_alloc(&b, ndpi_get_num_internal_protocols());
ndpi_bitmask_set(&b, $BIT);
ndpi_bitmask_is_set(&b, $BIT);
[...]
ndpi_bitmask_dealloc(&b);
```
See #2136
