Users can now specify a custom libpcap installation path on Linux using
--with-libpcap=PATH, enabling testing with different libpcap versions
without system-wide installation. The implementation prefers static
libraries, auto-detects dependencies via pkg-config, and displays the
selected libpcap path and libraries in the configuration summary.
🤖 Generated with [Claude Code](https://claude.com/claude-code)
Co-authored-by: Claude Sonnet 4.5 <noreply@anthropic.com>
The issue about `config.txt` files is that they contains paths:
* to configuration files, which are in the source tree
* to the dynamic plugins, which are in the build tree
Solution:
* copy all configuration files into the build tree
* all those paths are about the build tree
* tests run from the build tree, no from the source tree anymore
The configure flags --disable-shared and --disable-static were properly
recognized by libtool but ignored by nDPI's custom src/lib/Makefile.in,
which always built both static and shared libraries regardless of the
flags specified.
This commit fixes the issue by:
1. Exporting enable_shared and enable_static variables from configure.ac
via AC_SUBST so they're available in Makefiles
2. Adding configure-time error checks:
- Prevent both --disable-shared and --disable-static simultaneously
- Require static library for --enable-fuzztargets (fuzz targets need
static linking for proper instrumentation)
3. Modifying src/lib/Makefile.in to conditionally build libraries
4. Updating all build targets to support dynamic linking when static
library is disabled.
These targets now:
- Use static library when available (preferred, default behavior)
- Fall back to dynamic linking with -lndpi when --disable-static
5. Adding configuration summary output showing which libraries will be
built (enabled/disabled status for both shared and static)
fuzz: disable creation of (unused) shared library
🤖 Generated with [Claude Code](https://claude.com/claude-code)
Co-authored-by: Claude <noreply@anthropic.com>
- always use `-Wextra` compilation flag; it was already used in CI
- always compile `ndpiSimpleIntegration` when building examples
- don't mess with optimization flags: `CFLAGS` default value is "-g -O2"
and the user can change it
Try to test -O1,2,3,s flags in CI.
Fix some warnings.
Display a comprehensive configuration summary after ./configure completes,
showing:
- Package information (version, API version, git date)
- Installation paths
- Build configuration (compilers, debug mode, sanitizers, coverage, LTO)
- Core features (PCRE2, nBPF, libgcrypt, global context, TLS sigs, CRoaring)
- Optional dependencies (MaxMindDB, JSON-C, RRDtool, libnuma, gperftools)
- Build targets (library-only mode, examples, unit tests, fuzz targets, DPDK)
- Compiler flags (CFLAGS, NDPI_CFLAGS, LDFLAGS, NDPI_LDFLAGS, libs)
The summary is built dynamically as a single string variable and output with
one AC_MSG_NOTICE call, resulting in clean output without "configure:" prefix
on every line.
🤖 Generated with [Claude Code](https://claude.com/claude-code)
Co-authored-by: Claude <noreply@anthropic.com>
This commit improves cross-compilation support by replacing runtime
uname calls with autotools host detection. Changes include:
- Add AC_CANONICAL_HOST to detect build/host/target systems
- Replace uname-based OS detection with $host_os checks
- Use AC_CHECK_TOOL for AR and RANLIB (cross-compilation aware)
- Set MACHINE from $host_cpu instead of uname -m
- Remove ARM-specific libnuma exclusion (let configure detect)
- Export OS_TYPE to Makefiles for consistent platform checks
- Stop overriding CC in Makefiles (respects configure settings)
🤖 Generated with [Claude Code](https://claude.com/claude-code)
Co-authored-by: Claude <noreply@anthropic.com>
Fix improper handling of CFLAGS and LDFLAGS throughout the build system.
Also remove hardcoded debug flags that prevented production builds
without symbols.
Problems:
---------
1. CFLAGS/LDFLAGS handling:
The build system was using `CFLAGS +=` and `LDFLAGS +=` to append
package-specific flags, which modifies the user's environment variables
instead of keeping package and user flags separate. This caused:
- User-specified optimization levels being overridden by package defaults
- Inability to properly override flags at configure or make time
- Problems with cross-compilation and embedded toolchains
2. Hardcoded -g flags:
Debug symbols (-g) were hardcoded in several Makefiles, forcing debug
symbols in all builds including production. This caused:
- Larger binary sizes (library and tools)
- No way to build without debug symbols
- Conflicts with user's debug level preferences (-g1, -g2, -g3)
- Redundancy with configure options (--enable-debug-build)
Solutions:
----------
1. Implement proper CFLAGS/LDFLAGS separation using AM_CFLAGS/AM_LDFLAGS:
- Added `CFLAGS = @CFLAGS@` to preserve configure-time flags
- Added `LDFLAGS = @LDFLAGS@` to preserve configure-time flags
- Changed `CFLAGS +=` to `AM_CFLAGS =` and `AM_CFLAGS +=`
- Changed `LDFLAGS +=` to `AM_LDFLAGS =` and `AM_LDFLAGS +=`
- Updated compilation rules: $(CC) $(AM_CFLAGS) $(CFLAGS) ...
- Updated linking rules: $(CC) ... $(AM_LDFLAGS) $(LDFLAGS) ...
2. Remove all hardcoded -g flags from Makefiles:
- Debug symbols now controlled via configure (--enable-debug-build)
or user CFLAGS (e.g., CFLAGS="-g3")
Flag ordering ensures:
- Package flags come first (e.g., -O2, -fPIC)
- User flags come after and can override (e.g., -O3)
- Last flag wins for conflicting options
🤖 Generated with [Claude Code](https://claude.com/claude-code)
Co-authored-by: Claude <noreply@anthropic.com>
This commit implements comprehensive improvements to the nDPI build system
to enhance portability, enable parallel testing, and ensure reliable
out-of-tree (VPATH) builds across all platforms.
Changes:
1. Optimize library linking order (configure.ac, all Makefiles)
- Reorder ADDITIONAL_LIBS to follow proper dependency hierarchy
- Move low-level libraries (libm) to end of link line
- Ensures compatibility with --as-needed linker flag
- Improves LTO and static linking support
2. Fix VPATH build dependencies (all Makefiles)
- Add explicit dependencies on generated headers (ndpi_config.h, ndpi_define.h)
- Prevents race conditions in parallel builds (make -j)
- Ensures headers exist before compilation starts
3. Replace mkdir -p with portable $(MKDIR_P) macro
4. Enable parallel test execution (configure.ac)
- Add 'parallel-tests' option to AM_INIT_AUTOMAKE
- Allows test suites to run concurrently during 'make check'
5. Add defensive .NOTPARALLEL directive (Makefile.am)
- Prevents race conditions if 'make -j clean distclean' is run
6. Fix clean target completeness (src/lib/Makefile.in)
- Remove all .so symlinks (libndpi.so, libndpi.so.N)
- Add cleanup for Windows DLL files (*.dll)
- Explicitly remove versioned shared libraries
🤖 Generated with [Claude Code](https://claude.com/claude-code)
Co-Authored-By: Claude <noreply@anthropic.com>
Co-Authored-By: Ivan Nardi <nardi.ivan@gmail.com>
- Fixed some typos and inconsistent option names in error messages
- Improved Git Detection in configure script
- Added informative warnings when optional dependencies are missing
- Improve error handling on autogen.sh script
- Simplify fuzzing Makefile, creating common FUZZ_LINK_COMMAND template for all fuzz targets
- Added *_DEPENDENCIES declarations for fuzz targets with dictionaries
- Implemented incremental corpus building to avoid unnecessary rebuilds
🤖 Generated with [Claude Code](https://claude.com/claude-code)
Co-authored-by: Claude <noreply@anthropic.com>
Initial work to support out-of-tree builds
```
./autogen.sh
mkdir build
cd build
../configure
make
make check
```
IMPORTANT: `autogen.sh` doesn't call `configure` automatically anymore!!
You have to do: `./autogen.sh && ./configure --$OPTIONS`.
A little bit annoying but the pattern `autogen && configure && make` is
very common on Linux.
Known issues:
* `make doc` doesn't work in out-of-tree builds, yet
* Windows/MinGW/DPDK (out-of-tree) builds have not been tested, so it is unlikely they work
See: #2992
* configure: improve roaring version detection
Replace GCC version heuristic with proper C11 atomics feature detection.
Previously, the configure script used GCC version >= 7 as a proxy to
determine whether to use roaring v4 or fall back to the old version.
This approach had several limitations:
- Only worked reliably with GCC
- Didn't verify actual C11 support
- Could fail with other compilers (Clang, ICC, etc.)
Roaring v4 requires C11 atomics (stdatomic.h, _Atomic, etc.) as per
roaring.h:547. This commit implements a proper feature test using
AC_COMPILE_IFELSE that checks:
- C11 standard support (__STDC_VERSION__ >= 201112L)
- C11 atomics not disabled (__STDC_NO_ATOMICS__)
- Working <stdatomic.h> header
- Functional atomic operations (atomic_fetch_add_explicit, etc.)
Benefits:
- Works correctly with any C11-compliant compiler
- Tests actual requirements instead of compiler version
- More robust across different platforms
The --enable-old-croaring flag continues to work as before, allowing
users to force the old roaring version when needed.
On CI, we can now autodetect roaring version even with mingw compiler.
🤖 Generated with [Claude Code](https://claude.com/claude-code)
Co-Authored-By: Claude <noreply@anthropic.com>
Co-Authored-By: Ivan Nardi <nardi.ivan@gmail.com>
* Fix compilation with mingw compiler
Fix the warning:
```
third_party/src/roaring.c: In function ‘roaring64_bitmap_remove_bulk’:
third_party/src/roaring.c:24508:61: error: ‘leaf’ may be used uninitialized in this function [-Werror=maybe-uninitialized]
24508 | static inline uint64_t get_index(leaf_t leaf) { return leaf >> 8; }
| ~~~~~^~~~
third_party/src/roaring.c:25166:20: note: ‘leaf’ was declared here
25166 | leaf_t leaf;
| ^~~~
cc1: all warnings being treated as errors
```
---------
Co-authored-by: Claude <noreply@anthropic.com>
Old compilers (example: gcc < 7, mingw or VS older than 2022) don't
work with latest croaring versions; note that we had this issue even
before this change...
We somehow try to autodect if we can use latest version (TODO: we can do
better!): otherwise we fallback to legacy code.
The user can force that via `--enable-old-croaring` option: that's the
option to use if you get any errors on `roaring.c` filw while compiling.
Remove `-AC_COMPILE_IFELSE` check which does nothing and provides
misleading output!
On CI, we always use legacy version on Windows (if we use VisualStudio)
and with Mingw compiler.
Please, note that before the recent code adding
`NDPI_UNRESOLVED_HOSTNAME` support, the croaring code, even if present
in the repository, was NEVER used!!
Without the `-fsanitize-memory-track-origins` flag, MSAN job is ~30%
faster. Since this flag is useful only while debugging (and not to
simply discover memory issues), avoid it on the CI. Note that, by
default it is still enabled by default.
Right now, MingW runs on *every* ubuntu builds: limit it only to the
standard matrix (i.e. ubuntu 20.04, 22.04, 24.04 with default
configuration), without any sanitizers (note that MingW doesn't support
*san anyway).
armhf job is by far the longest job in the CI: remove asan configuration
to make it faster. Note that we already have a lot of different jobs (on
x86_64) with some sanitizers, and that the other 2 jobs on arm/s390x don't
have asan support anyway.
If we really, really want a job with arm + asan we can add it as a
async/scheduled job.
Remove an old workaround for ubuntu jobs
Avoid installing packages needed only for the documentation
About `check_symbols.sh` script: even if uses the compiled library/objects,
it basicaly only checks if we are using, in the source code, same functions
that we shoudn't. We don't need to perform the same kind of check so
many times..
* Added
size_t ndpi_compress_str(const char * in, size_t len, char * out, size_t bufsize);
size_t ndpi_decompress_str(const char * in, size_t len, char * out, size_t bufsize);
used to compress short strings such as domain names. This code is based on
https://github.com/Ed-von-Schleck/shoco
* Major code rewrite for ndpi_hash and ndpi_domain_classify
* Improvements to make sure custom categories are loaded and enabled
* Fixed string encoding
* Extended SalesForce/Cloudflare domains list
* Integrated RoaringBitmap v3
* Renamed ndpi_bitmap64 ro ndpi_bitmap64_fuse
* Fixes to ndpi_bitmap for new roaring library
* Fixes for bitmap serialization
* Fixed format
* Warning fix
* Conversion fix
* Warning fix
* Added check for roaring v3 support
* Updated file name
* Updated path
* Uses clang-9 (instead of clang-7) for builds
* Fixed fuzz_ds_bitmap64_fuse
* Fixes nDPI printf handling
* Disabled printf
* Yet another printf fix
* Cleaup
* Fx for compiling on older platforms
* Fixes for old compilers
* Initialization changes
* Added compiler check
* Fixes for old compilers
* Inline function is not static inline
* Added missing include
* `ndpi_typedefs.h`: requires to include `ndpi_config.h` for the `HAVE_STRUCT_TIMESPEC` check
That will never happen, because `USE_GLOBAL_CONTEXT` is defined inside `ndpi_config.h`.
It's better to use `CFLAGS` to achieve the same.
Signed-off-by: Toni Uhlig <matzeton@googlemail.com>
Add the concept of "global context".
Right now every instance of `struct ndpi_detection_module_struct` (we
will call it "local context" in this description) is completely
independent from each other. This provide optimal performances in
multithreaded environment, where we pin each local context to a thread,
and each thread to a specific CPU core: we don't have any data shared
across the cores.
Each local context has, internally, also some information correlating
**different** flows; something like:
```
if flow1 (PeerA <-> Peer B) is PROTOCOL_X; then
flow2 (PeerC <-> PeerD) will be PROTOCOL_Y
```
To get optimal classification results, both flow1 and flow2 must be
processed by the same local context. This is not an issue at all in the far
most common scenario where there is only one local context, but it might
be impractical in some more complex scenarios.
Create the concept of "global context": multiple local contexts can use
the same global context and share some data (structures) using it.
This way the data correlating multiple flows can be read/write from
different local contexts.
This is an optional feature, disabled by default.
Obviously data structures shared in a global context must be thread safe.
This PR updates the code of the LRU implementation to be, optionally,
thread safe.
Right now, only the LRU caches can be shared; the other main structures
(trees and automas) are basically read-only: there is little sense in
sharing them. Furthermore, these structures don't have any information
correlating multiple flows.
Every LRU cache can be shared, independently from the others, via
`ndpi_set_config(ndpi_struct, NULL, "lru.$CACHE_NAME.scope", "1")`.
It's up to the user to find the right trade-off between performances
(i.e. without shared data) and classification results (i.e. with some
shared data among the local contexts), depending on the specific traffic
patterns and on the algorithms used to balance the flows across the
threads/cores/local contexts.
Add some basic examples of library initialization in
`doc/library_initialization.md`.
This code needs libpthread as external dependency. It shouldn't be a big
issue; however a configure flag has been added to disable global context
support. A new CI job has been added to test it.
TODO: we should need to find a proper way to add some tests on
multithreaded enviroment... not an easy task...
*** API changes ***
If you are not interested in this feature, simply add a NULL parameter to
any `ndpi_init_detection_module()` calls.
Try using latest gcc and clang versions.
We still care about RHEL7: since handling a RHEL7 runner on GitHub is
quite complex, let try to use a similar version of gcc, at least
Move from PCRE to PCRE2. PCRE is EOL and won't receive any security
updates anymore. Convert to PCRE2 by converting any function PCRE2 new
API.
Also update every entry in github workflows and README to point to the
new configure flag. (--with-pcre2)
Signed-off-by: Christian Marangi <ansuelsmth@gmail.com>
Latest RoaringBitmap version (introduced with bf413afb) triggers a new
warning with GCC-7:
```
ivan@ivan-Latitude-E6540:~/svnrepos/nDPI(dev)$ CC=gcc-7 CXX=g++-7 ./autogen.sh && make -s
autoreconf: Entering directory `.'
[...]
third_party/src/roaring.c:1815:1: warning: ‘no_sanitize’ attribute directive ignored [-Wattributes]
static inline int array_container_cardinality(const array_container_t *array) {
^~~~~~
third_party/src/roaring.c:1964:5: warning: ‘no_sanitize’ attribute directive ignored [-Wattributes]
const array_container_t *container2) {
[..]
```
The core issue is that `no_sanitize` attribute is defined only for GCC
>= 8.
That breaks the CI since we still use GCC-7 and `-Werror`: add a simple
workaround.
Fix compilation on Windows
The goal of this fuzzer is to test init and deinit of the library, with
different configurations. In details:
* random memory allocation failures, even during init phase
* random `ndpi_init_prefs` parameter of `ndpi_init_detection_module()`
* random LRU caches sizes
* random bitmask of enabled protocols
* random parameters of `ndpi_set_detection_preferences()`
* random initialization of opportunistic TLS
* random load/don't load of configuration files
This new fuzzer is a C++ file, because it uses `FuzzedDataProvider`
class (see
https://github.com/google/fuzzing/blob/master/docs/split-inputs.md).
Note that the (existing) fuzzers need to be linked with C++ compiler
anyway, so this new fuzzer doesn't add any new requirements.
