Commit graph

5656 commits

Author SHA1 Message Date
Ivan Nardi
73d2c8657d
CI: update jobs (#3050) 2025-12-04 20:15:28 +01:00
Ivan Nardi
8b167cc25b
Build system: use proper autoconf variable for absolute paths (#3049) 2025-12-04 19:07:17 +01:00
Luca Deri
3f2f1f8ce4
Added ability to define protocol dissectors in shared libraries (#3047)
* Added ability to define protocol dissectors in shred libraries and load them at runtime

---------

Co-authored-by: Ivan Nardi <nardi.ivan@gmail.com>
2025-12-04 15:26:15 +01:00
Ivan Nardi
5cae544a40
s7comm: small fixes and extend tests (#3046) 2025-11-30 15:52:22 +01:00
Ivan Nardi
b7ef41e091 Revert "fuzz: temporary hack for introspector builds (#3041)"
This reverts commit 16916fa8bb.
2025-11-30 14:49:14 +01:00
Luca Deri
b6f0d08086 Added testing pcap files for EthernetIP 2025-11-29 11:38:55 +01:00
Ivan Nardi
b762509177
S7Comm: follow-up to complete monitoring feature (#3045) 2025-11-28 18:11:24 +01:00
Alfredo Cardigliano
9e980968d6 Extend s7comm dissector with metadata extractions (count requests by type) 2025-11-28 17:15:12 +01:00
Ivan Nardi
d76007054c
Build system: Fix --disable-shared and --disable-static flags being ignored (#3044)
The configure flags --disable-shared and --disable-static were properly
recognized by libtool but ignored by nDPI's custom src/lib/Makefile.in,
which always built both static and shared libraries regardless of the
flags specified.

This commit fixes the issue by:

1. Exporting enable_shared and enable_static variables from configure.ac
   via AC_SUBST so they're available in Makefiles

2. Adding configure-time error checks:
   - Prevent both --disable-shared and --disable-static simultaneously
   - Require static library for --enable-fuzztargets (fuzz targets need
     static linking for proper instrumentation)

3. Modifying src/lib/Makefile.in to conditionally build libraries

4. Updating all build targets to support dynamic linking when static
   library is disabled.
   These targets now:
   - Use static library when available (preferred, default behavior)
   - Fall back to dynamic linking with -lndpi when --disable-static

5. Adding configuration summary output showing which libraries will be
   built (enabled/disabled status for both shared and static)

fuzz: disable creation of (unused) shared library

🤖 Generated with [Claude Code](https://claude.com/claude-code)

Co-authored-by: Claude <noreply@anthropic.com>
2025-11-25 15:32:29 +01:00
Ivan Nardi
31060d1645
Build system: Remove libndpiReader.a intermediate archive (#3043)
Simplify the example/Makefile.in build process by removing the
libndpiReader.a static archive target. Instead, compile and link
all common object files directly into the ndpiReader executable.

🤖 Generated with [Claude Code](https://claude.com/claude-code)

Co-authored-by: Claude <noreply@anthropic.com>
2025-11-25 12:54:53 +01:00
Ivan Nardi
44ed6f8e91
Build system: Fix out-of-tree builds for fuzz targets (#3042)
Replace relative path references (../) with $(top_srcdir) in
fuzz/Makefile.am to properly support out-of-tree builds (VPATH builds).

🤖 Generated with [Claude Code](https://claude.com/claude-code)

Co-authored-by: Claude <noreply@anthropic.com>
2025-11-24 21:10:36 +01:00
Ivan Nardi
16916fa8bb
fuzz: temporary hack for introspector builds (#3041) 2025-11-24 20:25:10 +01:00
Ivan Nardi
b93c2a8dbc
Fix some warnings when compiling with -pedantic (#3040) 2025-11-24 19:47:11 +01:00
Ivan Nardi
b2357c29c3
Build system: Standardize and improve clean/distclean targets (#3039)
This commit improves the nDPI build system by standardizing cleanup
targets, improving portability, and ensuring complete removal of
generated files during `make distclean`.

Changes:

1. Standardize clean targets (replace /bin/rm with portable $(RM))

2. Add distclean-local targets for complete cleanup

3. Add missing clean/distclean targets

4. Remove obsolete commented-out curl detection code

5. fuzz/Makefile.am: Fix out-of-tree build compatibility by replacing hardcoded
   relative paths (../example/fuzz_*.o) with proper $(top_builddir) variables.
   Add distclean-local target.
2025-11-24 18:42:03 +01:00
Alfredo Cardigliano
3c279d33b7 Fix escape of special chars in ndpi_json_string_escape 2025-11-24 18:24:34 +01:00
Luca Deri
2a1bc86bca RPM fix 2025-11-23 11:04:15 +01:00
Ivan Nardi
19ee4f6c33
Build system: minor fixes about flag compilation and example dependencies (#3038)
- always use `-Wextra` compilation flag; it was already used in CI
- always compile `ndpiSimpleIntegration` when building examples
- don't mess with optimization flags: `CFLAGS` default value is "-g -O2"
  and the user can change it

Try to test -O1,2,3,s flags in CI.

Fix some warnings.
2025-11-21 15:51:29 +01:00
Ivan Nardi
4096d758bb
Build system: Add configuration summary at end of ./configure (#3037)
Display a comprehensive configuration summary after ./configure completes,
showing:
- Package information (version, API version, git date)
- Installation paths
- Build configuration (compilers, debug mode, sanitizers, coverage, LTO)
- Core features (PCRE2, nBPF, libgcrypt, global context, TLS sigs, CRoaring)
- Optional dependencies (MaxMindDB, JSON-C, RRDtool, libnuma, gperftools)
- Build targets (library-only mode, examples, unit tests, fuzz targets, DPDK)
- Compiler flags (CFLAGS, NDPI_CFLAGS, LDFLAGS, NDPI_LDFLAGS, libs)

The summary is built dynamically as a single string variable and output with
one AC_MSG_NOTICE call, resulting in clean output without "configure:" prefix
on every line.

🤖 Generated with [Claude Code](https://claude.com/claude-code)

Co-authored-by: Claude <noreply@anthropic.com>
2025-11-20 22:11:39 +01:00
Ivan Nardi
af1d80605c
Build system: Fix incomplete .inc file dependency tracking (#3036)
Fix missing dependencies for .inc files included by ndpi_main.c and
ndpi_fingerprint.c.

🤖 Generated with [Claude Code](https://claude.com/claude-code)

Co-authored-by: Claude <noreply@anthropic.com>
2025-11-20 19:41:20 +01:00
Ivan Nardi
59d9135751
Build system: Add proper cross-compilation support (#3035)
This commit improves cross-compilation support by replacing runtime
uname calls with autotools host detection. Changes include:

- Add AC_CANONICAL_HOST to detect build/host/target systems
- Replace uname-based OS detection with $host_os checks
- Use AC_CHECK_TOOL for AR and RANLIB (cross-compilation aware)
- Set MACHINE from $host_cpu instead of uname -m
- Remove ARM-specific libnuma exclusion (let configure detect)
- Export OS_TYPE to Makefiles for consistent platform checks
- Stop overriding CC in Makefiles (respects configure settings)

🤖 Generated with [Claude Code](https://claude.com/claude-code)

Co-authored-by: Claude <noreply@anthropic.com>
2025-11-20 15:06:41 +01:00
Ivan Nardi
e49e93cc17
Build system: Respect user CFLAGS and LDFLAGS, remove hardcoded -g (#3034)
Fix improper handling of CFLAGS and LDFLAGS throughout the build system.
Also remove hardcoded debug flags that prevented production builds
without symbols.

Problems:
---------
1. CFLAGS/LDFLAGS handling:
   The build system was using `CFLAGS +=` and `LDFLAGS +=` to append
   package-specific flags, which modifies the user's environment variables
   instead of keeping package and user flags separate. This caused:
   - User-specified optimization levels being overridden by package defaults
   - Inability to properly override flags at configure or make time
   - Problems with cross-compilation and embedded toolchains

2. Hardcoded -g flags:
   Debug symbols (-g) were hardcoded in several Makefiles, forcing debug
   symbols in all builds including production. This caused:
   - Larger binary sizes (library and tools)
   - No way to build without debug symbols
   - Conflicts with user's debug level preferences (-g1, -g2, -g3)
   - Redundancy with configure options (--enable-debug-build)

Solutions:
----------
1. Implement proper CFLAGS/LDFLAGS separation using AM_CFLAGS/AM_LDFLAGS:
   - Added `CFLAGS = @CFLAGS@` to preserve configure-time flags
   - Added `LDFLAGS = @LDFLAGS@` to preserve configure-time flags
   - Changed `CFLAGS +=` to `AM_CFLAGS =` and `AM_CFLAGS +=`
   - Changed `LDFLAGS +=` to `AM_LDFLAGS =` and `AM_LDFLAGS +=`
   - Updated compilation rules: $(CC) $(AM_CFLAGS) $(CFLAGS) ...
   - Updated linking rules: $(CC) ... $(AM_LDFLAGS) $(LDFLAGS) ...

2. Remove all hardcoded -g flags from Makefiles:
   - Debug symbols now controlled via configure (--enable-debug-build)
     or user CFLAGS (e.g., CFLAGS="-g3")

Flag ordering ensures:
- Package flags come first (e.g., -O2, -fPIC)
- User flags come after and can override (e.g., -O3)
- Last flag wins for conflicting options

🤖 Generated with [Claude Code](https://claude.com/claude-code)

Co-authored-by: Claude <noreply@anthropic.com>
2025-11-20 12:43:11 +01:00
Ivan Nardi
1bbafbd5d2 Build system: Apply warning flags consistently to performance tests
Ensure that performance test tools are compiled with project warning flags
(NDPI_CFLAGS) for consistent code quality standards across the codebase.

Previously, tests/performance/Makefile.in compiled tools (gcrypt-int,
gcrypt-gnu, substringsearch, strnstr, geo, patriciasearch) with only
user-provided CFLAGS, missing the project's warning flags (-W, -Wall,
-Wno-address-of-packed-member).

🤖 Generated with [Claude Code](https://claude.com/claude-code)

Co-Authored-By: Claude <noreply@anthropic.com>
2025-11-18 21:08:53 +01:00
Ivan Nardi
9587059598 Build system: Improve portability, parallelization, and VPATH builds
This commit implements comprehensive improvements to the nDPI build system
to enhance portability, enable parallel testing, and ensure reliable
out-of-tree (VPATH) builds across all platforms.

Changes:

1. Optimize library linking order (configure.ac, all Makefiles)
   - Reorder ADDITIONAL_LIBS to follow proper dependency hierarchy
   - Move low-level libraries (libm) to end of link line
   - Ensures compatibility with --as-needed linker flag
   - Improves LTO and static linking support

2. Fix VPATH build dependencies (all Makefiles)
   - Add explicit dependencies on generated headers (ndpi_config.h, ndpi_define.h)
   - Prevents race conditions in parallel builds (make -j)
   - Ensures headers exist before compilation starts

3. Replace mkdir -p with portable $(MKDIR_P) macro

4. Enable parallel test execution (configure.ac)
   - Add 'parallel-tests' option to AM_INIT_AUTOMAKE
   - Allows test suites to run concurrently during 'make check'

5. Add defensive .NOTPARALLEL directive (Makefile.am)
   - Prevents race conditions if 'make -j clean distclean' is run

6. Fix clean target completeness (src/lib/Makefile.in)
   - Remove all .so symlinks (libndpi.so, libndpi.so.N)
   - Add cleanup for Windows DLL files (*.dll)
   - Explicitly remove versioned shared libraries

🤖 Generated with [Claude Code](https://claude.com/claude-code)

Co-Authored-By: Claude <noreply@anthropic.com>
Co-Authored-By: Ivan Nardi <nardi.ivan@gmail.com>
2025-11-18 21:08:53 +01:00
Ivan Nardi
6bb0585a27
fuzz: minor improvements. Be sure the right configuration is always loaded (#3031) 2025-11-18 14:11:16 +01:00
Ivan Nardi
e58f23dc75 tests: extend utests 2025-11-18 13:32:14 +01:00
Ivan Nardi
aa544dd826
croaring: update to 4.4.2 (from 4.3.6) (#3030) 2025-11-18 11:28:37 +01:00
Ivan Nardi
334cc23a8e
Build system improvements and fixes (#3026)
- Fixed some typos and inconsistent option names in error messages
- Improved Git Detection in configure script
- Added informative warnings when optional dependencies are missing
- Improve error handling on autogen.sh script
- Simplify fuzzing Makefile, creating common FUZZ_LINK_COMMAND template for all fuzz targets
- Added *_DEPENDENCIES declarations for fuzz targets with dictionaries
- Implemented incremental corpus building to avoid unnecessary rebuilds

🤖 Generated with [Claude Code](https://claude.com/claude-code)

Co-authored-by: Claude <noreply@anthropic.com>
2025-11-18 10:32:04 +01:00
Ivan Nardi
bf120e2aad
tests: improve parallelization (#3027)
Allow multiple configuration to run in parallel.
Allow fast failure
2025-11-18 10:31:50 +01:00
Luca Deri
cc9dc4c3f9 Moved to 5.1.0 2025-11-17 09:11:23 +01:00
Ivan Nardi
75db1a8a66 Fix ndpi_detection_process_packet() return value when the flow is already classified 2025-11-14 10:11:20 +01:00
Ivan Nardi
1b6db29080 fuzz: fix 2025-11-13 14:14:33 +01:00
Ivan Nardi
173f718f5a
fuzz: another attempt to fix introspector builds (#3025) 2025-11-12 19:57:14 +01:00
Ivan Nardi
14dc8eb1bd utests: small fix 2025-11-12 19:13:18 +01:00
Luca Deri
1cdb50e149 Added reference to https://github.com/ntop/nDPI/pull/3022 2025-11-09 19:41:46 +01:00
Luca Deri
5c327aafa0
Added nDPI Configuration Export (#3022)
* In order to reduce ndpi_main.c file size:
- Removed nDPI configuration code from ndpi_main.c and placed into ndpi_config.c
- Moved some utils functions from ndpi_main.c to ndpi_utils.c

* Added
- ndpi_dump_host_based_protocol_id()
- ndpi_dump_host_based_category_id()

to enable users to dump protocolId and categoryId of host-based protocols

ndpiReader
- Added
--protos-dump <mode>       | Dump host-based protocolId (mode=1) and categoryId (mode=2)
2025-11-09 19:39:47 +01:00
Ivan Nardi
e7737bb578
fuzz: add fuzzer to test ranking algorithm (#3024) 2025-11-09 16:49:35 +01:00
Ivan Nardi
e22a434709
Rework API to set custom memory allocator functions (#3023)
Full accounting of memory used by the library.

Change `ndpi_realloc()` prototype to be compatible with standard
`realloc()`.
Be compatible with croaring allocation logic.

Note that aligned allocations are used only by croaring code.
Note that flow allocations are used only by the application, not by the
library.

API changes:
* remove `set_ndpi_malloc()` and `set_ndpi_free()`; use
  `ndpi_set_memory_alloction_functions()` instead
2025-11-09 13:11:55 +01:00
Luca Deri
bb10ecc380 RDP: Added check to detect probing attempts 2025-11-06 22:45:19 +01:00
Luca Deri
256ccc1cac Added check to void false positives 2025-11-06 22:44:45 +01:00
Ivan Nardi
c37937a211
fuzz: improve fuzzing coverage (#3020)
We should pay attention to tell ndpiReader configuration files and
libnDPI configuration files!! Better solution?

Be sure that configuration files are located where they are expected.
In oss-fuzz enviroment we can't make any assumptions about the current
working directory of your fuzz target.
2025-11-04 21:04:29 +01:00
Ivan Nardi
f94ce7d9d4
Fix classification state for flows in monitoring (#3019) 2025-11-04 10:46:14 +01:00
Ivan Nardi
433f708951
Fix compilation when using external libgcrypt (#3018)
ndpiReader: fix encodeDomainsUnitTest test
2025-11-04 10:41:00 +01:00
Ivan Nardi
be9473f7b4 Update documentation
[skip ci]
2025-11-03 19:55:13 +01:00
Ivan Nardi
6ff4eff377
Update CHANGELOG.md 2025-11-03 19:54:28 +01:00
Ivan Nardi
813e074a88
Update CHANGELOG.md [WIP]
[no-ci]
2025-11-03 18:21:57 +01:00
Ivan Nardi
797edd6c43 Update .gitignore 2025-11-03 13:01:09 +01:00
Ivan Nardi
bb4c6b0a3a
Update every lists (#3017) 2025-11-03 13:00:27 +01:00
Ivan Nardi
a9e38cc504 ndpiReader: fix typo
Credits to @s4n-cz.
Close #3015
2025-11-03 12:36:12 +01:00
Ivan Nardi
83d85775a8
Provide an explicit state for the flow classification process (#2942)
Application should keep calling nDPI until flow state became
`NDPI_STATE_CLASSIFIED`.

The main loop in the application is simplified to something like:
```
res = ndpi_detection_process_packet(...);
if(res->state == NDPI_STATE_CLASSIFIED) {
  /* Done: you can get finale classification and all metadata.
     nDPI doesn't need more packets for this flow */
} else {
  /* nDPI needs more packets for this flow. The provided
     classification is not final and more metadata might be
     extracted.
     If `res->state` is `NDPI_STATE_PARTIAL`, partial/initial
     classification is available in `res->proto`
     as usual but it can be updated later.
  */
}

/*
    Example A (QUIC flow):
     pkt 1: proto QUIC state NDPI_STATE_PARTIAL
     pkt 2: proto QUIC/Youtube  state NDPI_STATE_CLASSIFIED
    Example B (GoogleMeet call):
     pkt 1:   proto STUN state NDPI_STATE_PARTIAL
     pkt N:   proto DTLS state NDPI_STATE_PARTIAL
     pkt N+M: proto DTLS/GoogleCall state NDPI_STATE_CLASSIFIED
    Example C (standard TLS flow):
     pkt 1:   proto Unknown state NDPI_STATE_INSPECTING
     pkt 2:   proto Unknown state NDPI_STATE_INSPECTING
     pkt 3:   proto Unknown state NDPI_STATE_INSPECTING
     pkt 4:   proto TLS/Facebook state NDPI_STATE_PARTIAL
     pkt N:   proto TLS/Facebook state NDPI_STATE_CLASSIFIED
 */
}
```
You can take a look at `ndpiReader` for a slightly more complex example.

API changes:
* remove the third parameter from `ndpi_detection_giveup()`. If you need
to know if the classification flow has been guessed, you can access
`flow->protocol_was_guessed`
* remove `ndpi_extra_dissection_possible()`
* change some prototypes from accepting `ndpi_protocol foo` to
`ndpi_master_app_protocol bar`. The update is trivial: from `foo` to
`foo.proto`
2025-11-03 12:08:15 +01:00
Ivan Nardi
6ab338928c
Add support for out-of-tree builds (#2993)
Initial work to support out-of-tree builds
```
./autogen.sh
mkdir build
cd build
../configure
make
make check
```
IMPORTANT: `autogen.sh` doesn't call `configure` automatically anymore!!

You have to do: `./autogen.sh && ./configure --$OPTIONS`.
A little bit annoying but the pattern `autogen && configure && make` is
very common on Linux.

Known issues:
* `make doc` doesn't work in out-of-tree builds, yet
* Windows/MinGW/DPDK (out-of-tree) builds have not been tested, so it is unlikely they work

See: #2992
2025-11-03 11:58:59 +01:00