Both Windows downloads come from npcap.com, which periodically refuses the TLS
handshake. A single failed attempt failed the job, and the failures differed
run to run, so they looked like a problem with whatever change was under test.
The SDK fetch in build_windows.sh checked nothing. wget wrote into a directory
with -P, so a partial download could be left behind and a later attempt would
create a .1 file while unzip kept reading the corrupt original. Neither the
wget nor the unzip result was inspected and the script sets no -e, so a failed
fetch continued and resurfaced hundreds of lines later as
fatal error C1083: Cannot open include file: 'pcap.h'
It now retries, writes to a fixed path, fails explicitly when every attempt is
exhausted, and validates both files the build actually consumes: Include/pcap.h
and Lib/x64/wpcap.lib, the latter being linked by engine_build.py. The call
site propagates the failure, which subprocess.check_call surfaces.
The OEM installer download in the two workflows gets the same treatment. Its
exit status is now inspected as well: Start-Process -Wait does not fail when
the child exits nonzero, so a failed installation was previously indistinguish-
able from a successful one. The requests also address npcap.com directly rather
than relying on the npcap.org redirect, and the credentials are passed through
the environment instead of being interpolated into the script, so characters
that are significant to PowerShell cannot alter it.
Retries make transient failures survivable and sustained outages legible. They
do not help pull requests from forks, where the credentials are unavailable and
the download cannot succeed at all.
Consecutive merges each start a full matrix, and without a concurrency group
the older runs keep competing for runners even though the newer commit already
contains their changes. When multiple merges land within minutes, the
superseded runs have to be cancelled by hand, and manual cancellation risks
cancelling the newest run instead.
Grouping by workflow and ref lets GitHub cancel the superseded run
automatically.
Applied to the three build_test workflows only. build_wheel_publish is
excluded because release and publication jobs are not supersedable and must run
to completion. cifuzz and codeql are unchanged; this is scoped to the three OS
build matrices where the duplication occurs.
Windows spawns meter processes instead of forking them, so plugin instances
must be pickled to reach them. The plugin was defined inside the test
function, which cannot be pickled, and the Windows job failed with
AttributeError: Can't pickle local object
'NFStreamTest.test_anonymize_absent_values.<locals>.MixedValues'
Linux and macOS fork, so the plugin was inherited rather than pickled and the
test passed there. The failure was also masked at first because the Windows
jobs were failing earlier, at the Npcap download, and never reached the tests.
Moving the class to module level makes it picklable, matching how the other
plugins used in the suite are defined.
The pip cache step captured pythonLocation, the interpreter installation
itself. Every run has setup-python freshly provision the interpreter and
install its own pip, and actions/cache then restored the saved tree over that
fresh install. When the two pip versions differed, the overlay produced a
mixed installation.
That is what broke macOS PyPy. The run that created the cache installed pip
24.0 via setup-python, upgraded it to 26.1.2, then built and tested
successfully, so pip was healthy when saved. Later runs provisioned pip 24.0
again and restored the 26.1.2 tree over it, leaving a newer
build_env/installer.py importing get_runnable_pip from an older
utils/misc.py. Every run then failed inside pip before NFStream was reached,
across unrelated pull requests, until the cache was deleted by hand.
setup-python caches pip's own download and wheel cache instead, which is inert
data rather than an installation, so nothing is overlaid onto the interpreter.
pip reuses wheels it built from source distributions, so the expensive builds
that motivated the original caching, pandas on PyPy in particular, are still
avoided; only the install step is repeated.
Requirements are now installed on every run rather than skipped on a cache
hit, so the environment always matches dev_requirements.txt. The explicit pip
upgrade is dropped as unnecessary.
The codecov PyPI package is installed by the four workflows that install
development requirements but is never invoked. Coverage is uploaded by the
codecov/codecov-action GitHub Action, and codecov.yml configures the Codecov
service rather than this package. The PyPI uploader has also been deprecated
upstream in favour of the action and the Codecov CLI.
NFAnonymizer guarded only on None, so an empty string was hashed like real
data. Any optional field that was never populated became a 128 character
blake2b digest, and because the key is fixed for the duration of an export
every such flow received the same digest. A consumer then sees a large
cohort apparently sharing one fingerprint, user agent or server name that
does not exist, and cannot distinguish it from a genuinely shared value.
This affects to_csv as well as to_pandas.
Absent values are now left untouched. Only str is compared against the
empty string: a truthiness test would stop anonymizing 0 and False, which
are legitimate values, and an unguarded equality would raise on a numpy
array stored by a plugin, whose comparison does not yield a bool.
String fields decoded from C via ffi.string().decode() returned ""
when the underlying C char array was empty (e.g. application_name
when protocol is not detected). This was inconsistent with non-sync
mode where the same fields are explicitly set to None, and caused a
subtle pandas bug: DataFrame built from NFlow had no nulls, but after
to_csv()/read_csv() round-trip NaN values appeared.
* Upgrade nDPI from 4.7.0 to 5.0 with Windows build fix (#230)
- Upgrade nDPI submodule from 4.7.0 to 5.0
- Fix all 9 nDPI 5.0 API breaking changes:
* Protocol structure: .app_protocol -> .proto.app_protocol
* Protocol structure: .master_protocol -> .proto.master_protocol
* ndpi_protocol2name() signature change
* ndpi_detection_giveup() signature change (removed enable_guess/guessed params)
* ndpi_extra_dissection_possible() replaced with state check
* ndpi_init_detection_module() now expects NULL
* Protocol bitmask removed (all protocols enabled by default)
* TCP/UDP struct size checks removed
* Fingerprint extraction: ja3_client -> ja4_client
- Maintain backward compatibility by keeping field names:
* client_fingerprint: Now JA4 (TLS), HASSH client (SSH), DHCP fingerprint
* server_fingerprint: Now JA3S (TLS), HASSH server (SSH)
- Update build system for nDPI 5.0:
* Add --with-only-libndpi flag to build.sh
* Make CFFI marker extraction optional in engine_build.py
- Update tests for nDPI 5.0:
* Update fingerprint assertions (JA3 MD5 -> JA4 format)
* Regenerate all 228 baseline test results
* Remove invalid test file (memcached.cap)
- All 29 tests passing
* Fix memcpy buffer overreads and enable DNS subclassification
- Fix buffer overread in fingerprint memcpy calls by using source size
instead of destination size:
- HASSH client/server: 33 bytes (was reading 48)
- JA4 client: 37 bytes (was reading 48)
- JA3 server: 33 bytes (was reading 48)
- DHCP: 48 bytes (consistent with others)
- Enable DNS subclassification (disabled by default in nDPI 5.0)
to restore detection of DNS.Apple, DNS.Google, etc.
- Regenerate test baselines with fixes applied
* Replace outdated Steam test pcaps with nDPI 5.0 version
- Remove steam.pcap and steam_datagram_relay_ping.pcapng (obsolete)
- Add steam.pcapng from nDPI 5.0 test suite
- nDPI 5.0 reworked Steam detection (ntop/nDPI#2264)
* Fix Windows build: add missing configure step for nDPI 5.0
The Windows build script was missing the ./configure --with-only-libndpi
step before make, causing nDPI headers to not be properly installed.
This resulted in the CFFI preprocessing marker //CFFI.NDPI_PACKED_STRUCTURES
being absent from ndpi_cdefinitions_packed.h, crashing engine_build.py.
Aligns build_windows.sh with build.sh which already had this step.
* Fix Windows build: install libpcap-dev for nDPI 5.0 configure
nDPI 5.0 configure requires libpcap when --with-only-libndpi is used
on MinGW (see ntop/nDPI#3114). Install mingw-w64-x86_64-libpcap via
pacman to satisfy this dependency.
* Upgrade nDPI from 4.7.0 to 5.0 with backward compatibility
- Upgrade nDPI submodule from 4.7.0 to 5.0
- Fix all 9 nDPI 5.0 API breaking changes:
* Protocol structure: .app_protocol -> .proto.app_protocol
* Protocol structure: .master_protocol -> .proto.master_protocol
* ndpi_protocol2name() signature change
* ndpi_detection_giveup() signature change (removed enable_guess/guessed params)
* ndpi_extra_dissection_possible() replaced with state check
* ndpi_init_detection_module() now expects NULL
* Protocol bitmask removed (all protocols enabled by default)
* TCP/UDP struct size checks removed
* Fingerprint extraction: ja3_client -> ja4_client
- Maintain backward compatibility by keeping field names:
* client_fingerprint: Now JA4 (TLS), HASSH client (SSH), DHCP fingerprint
* server_fingerprint: Now JA3S (TLS), HASSH server (SSH)
- Update build system for nDPI 5.0:
* Add --with-only-libndpi flag to build.sh
* Make CFFI marker extraction optional in engine_build.py
- Update tests for nDPI 5.0:
* Update fingerprint assertions (JA3 MD5 -> JA4 format)
* Regenerate all 228 baseline test results
* Remove invalid test file (memcached.cap)
- All 29 tests passing
* Fix memcpy buffer overreads and enable DNS subclassification
- Fix buffer overread in fingerprint memcpy calls by using source size
instead of destination size:
- HASSH client/server: 33 bytes (was reading 48)
- JA4 client: 37 bytes (was reading 48)
- JA3 server: 33 bytes (was reading 48)
- DHCP: 48 bytes (consistent with others)
- Enable DNS subclassification (disabled by default in nDPI 5.0)
to restore detection of DNS.Apple, DNS.Google, etc.
- Regenerate test baselines with fixes applied
* Replace outdated Steam test pcaps with nDPI 5.0 version
- Remove steam.pcap and steam_datagram_relay_ping.pcapng (obsolete)
- Add steam.pcapng from nDPI 5.0 test suite
- nDPI 5.0 reworked Steam detection (ntop/nDPI#2264)
* Fix Windows build: add missing configure step for nDPI 5.0
The Windows build script was missing the ./configure --with-only-libndpi
step before make, causing nDPI headers to not be properly installed.
This resulted in the CFFI preprocessing marker //CFFI.NDPI_PACKED_STRUCTURES
being absent from ndpi_cdefinitions_packed.h, crashing engine_build.py.
Aligns build_windows.sh with build.sh which already had this step.
pandas 3.0.0 (released Jan 21, 2026) segfaults on import when running
under PyPy. Cap pandas to <3 for PyPy installations while allowing
any version for CPython using environment markers.
- Replace manual string concatenation with csv.writer() for RFC 4180 compliance
- Update open_file() to use text mode with proper encoding and newline handling
- Remove manual quoting from csv_converter() as csv.writer() handles this properly
This addresses potential data corruption with special characters (commas, quotes,
newlines) in network flow data by using Python's standard CSV module instead
of manual string formatting.
Make IPv6 ip_size calculation consistent with IPv4 for unified accounting modes.
IPv6 ip6_un1_plen field excludes the IPv6 header (per RFC), while IPv4 tot_len
includes the IPv4 header. This caused inconsistent accounting behavior between
IPv4 and IPv6 flows.
Updated packet_get_ipv6_info() to include IPv6 header size in total length
calculation for consistent accounting semantics:
- Before: iph.tot_len = iph6->ip6_hdr.ip6_un1_plen (payload only)
- After: iph.tot_len = htons(sizeof(struct ndpi_ipv6hdr) + ntohs(iph6->ip6_hdr.ip6_un1_plen))
This ensures unified accounting behavior across IPv4 and IPv6:
- Mode 0: Raw packet (all headers + data)
- Mode 1: IP packet (IP header + transport + data)
- Mode 2: Transport segment (transport header + data)
- Mode 3: Application payload (data only)
Previously IPv6 Mode 1 and Mode 2 were identical, now they differ by 40 bytes.
