vrr/nDPI
2
0
Fork 0
mirror of https://github.com/vel21ripn/nDPI.git synced 2026-04-28 15:09:47 +00:00
Code Issues Projects Releases Packages Wiki Activity Actions 5
5656 commits 9 branches 12 tags 263 MiB
Commit graph

105 commits

Author SHA1 Message Date
Luca Deri
37ca034697 (C) update 2026-01-01 10:31:40 +01:00
Luca Deri
8aaff75ede
Fixes protocol inconsistencies (#3069) 
* Added code to address inconsistencies
* Added check for discarding dig errors
* Temporarily disabled whatsapp script: an update is required. See https://developers.facebook.com/docs/whatsapp/on-premises/sunset
 2025-12-31 20:54:35 +01:00
Luca Deri
3f2f1f8ce4
Added ability to define protocol dissectors in shared libraries (#3047) 
* Added ability to define protocol dissectors in shred libraries and load them at runtime

---------

Co-authored-by: Ivan Nardi <nardi.ivan@gmail.com>
 2025-12-04 15:26:15 +01:00
Ivan Nardi
b2357c29c3
Build system: Standardize and improve clean/distclean targets (#3039) 
This commit improves the nDPI build system by standardizing cleanup
targets, improving portability, and ensuring complete removal of
generated files during `make distclean`.

Changes:

1. Standardize clean targets (replace /bin/rm with portable $(RM))

2. Add distclean-local targets for complete cleanup

3. Add missing clean/distclean targets

4. Remove obsolete commented-out curl detection code

5. fuzz/Makefile.am: Fix out-of-tree build compatibility by replacing hardcoded
   relative paths (../example/fuzz_*.o) with proper $(top_builddir) variables.
   Add distclean-local target.
 2025-11-24 18:42:03 +01:00
Ivan Nardi
19ee4f6c33
Build system: minor fixes about flag compilation and example dependencies (#3038) 
- always use `-Wextra` compilation flag; it was already used in CI
- always compile `ndpiSimpleIntegration` when building examples
- don't mess with optimization flags: `CFLAGS` default value is "-g -O2"
  and the user can change it

Try to test -O1,2,3,s flags in CI.

Fix some warnings.
 2025-11-21 15:51:29 +01:00
Ivan Nardi
e22a434709
Rework API to set custom memory allocator functions (#3023) 
Full accounting of memory used by the library.

Change `ndpi_realloc()` prototype to be compatible with standard
`realloc()`.
Be compatible with croaring allocation logic.

Note that aligned allocations are used only by croaring code.
Note that flow allocations are used only by the application, not by the
library.

API changes:
* remove `set_ndpi_malloc()` and `set_ndpi_free()`; use
  `ndpi_set_memory_alloction_functions()` instead
 2025-11-09 13:11:55 +01:00
Ivan Nardi
6ab338928c
Add support for out-of-tree builds (#2993) 
Initial work to support out-of-tree builds
```
./autogen.sh
mkdir build
cd build
../configure
make
make check
```
IMPORTANT: `autogen.sh` doesn't call `configure` automatically anymore!!

You have to do: `./autogen.sh && ./configure --$OPTIONS`.
A little bit annoying but the pattern `autogen && configure && make` is
very common on Linux.

Known issues:
* `make doc` doesn't work in out-of-tree builds, yet
* Windows/MinGW/DPDK (out-of-tree) builds have not been tested, so it is unlikely they work

See: #2992
 2025-11-03 11:58:59 +01:00
Luca Deri
6001c9f34c Akamai IP address download 2025-10-18 12:25:16 +02:00
Luca Deri
d69446893d Added NDPI_MISMATCHING_PROTOCOL_WITH_IP flow risk 
Fixed host protocol matching
Added NDPI_PROTOCOL_AKAMAI protocol
 2025-10-17 23:48:44 +02:00
Ivan Nardi
17879f40dc
Automatically update the ip list of ChatGPT bots (#2977) 2025-10-02 20:18:09 +02:00
Ivan Nardi
113170cca4
New protocols for Amazon/AWS sub-classification (#2975) 
Add:
* Cognito
* API Gateway
* Kinesis
* EC2
* EMR
* S3
* Cloudfront
* DynamoDB

Keep `NDPI_PROTOCOL_AMAZON_AWS` for generic AWS traffic
 2025-10-02 11:48:25 +02:00
Luca Deri
3358fb5516 Code cleanup 2025-09-27 23:01:54 +02:00
Luca Deri
d20f4bb1c0 Added utility to print content of rank files 2025-09-12 11:27:53 +02:00
Ivan Nardi
30cd98f978
croaring: update to 4.3.6 (from 3.0.0) (#2934) 
Old compilers (example: gcc < 7, mingw or VS older than 2022) don't
work with latest croaring versions; note that we had this issue even
before this change...

We somehow try to autodect if we can use latest version (TODO: we can do
better!): otherwise we fallback to legacy code.
The user can force that via `--enable-old-croaring` option: that's the
option to use if you get any errors on `roaring.c` filw while compiling.

Remove `-AC_COMPILE_IFELSE` check which does nothing and provides
misleading output!

On CI, we always use legacy version on Windows (if we use VisualStudio)
and with Mingw compiler.

Please, note that before the recent code adding
`NDPI_UNRESOLVED_HOSTNAME` support, the croaring code, even if present
in the repository, was NEVER used!!
 2025-09-02 15:50:03 +02:00
Ivan Nardi
1dd6d4c30c
MS domain lists: avoid duplicates (#2928) 
Update all MS domains lists.

Close #2927
 2025-08-02 19:11:40 +02:00
Luca Deri
16f332eb41 Added simple tool hosts2domains used to extract domain names from hostnames written on a text file 
(one host per line)
 2025-07-21 19:02:25 +02:00
JH
ebb47c9a84
Add auto-updating cryptocurrency mining pool lists (#2891) 
Co-authored-by: Ivan Nardi <nardi.ivan@gmail.com>
 2025-07-01 20:27:22 +02:00
Luca Deri
38cc4ac22b Added IMO and Badoo files 2025-06-06 16:48:41 +02:00
Ivan Nardi
f0eb0ed1d8 Update url to download malicious sites 2025-06-05 14:26:25 +02:00
0xA50C1A1
b49b7eb45f Rename NDPI_PROTOCOL_UBUNTUONE protocol ID to NDPI_PROTOCOL_CANONICAL 2025-05-15 21:43:34 +02:00
Toni
470a479eb2
Remove ProtonVPN IP address fetch (#2811) 
* not available w/o an API token anymore

Signed-off-by: Toni Uhlig <matzeton@googlemail.com>
 2025-05-12 13:28:51 +02:00
Luca Deri
8d45dfaf73 Update Threema and VK ASN lists 2025-05-06 14:48:10 +02:00
Ivan Nardi
56ac5bf48b
Rework the old Starcraft code to identify traffic from generic Blizzard games (#2776) 
Remove `NDPI_PROTOCOL_STARCRAFT` and add a generic `NDPI_PROTOCOL_BLIZZARD`.
 2025-03-25 17:16:10 +01:00
Ivan Nardi
e2ed61524b
Avoid duplicated Microsoft domains (#2770) 
Update the list

Close #2767
 2025-03-18 11:56:50 +01:00
Ivan Nardi
678d284a25
TOR: fix ip lists (#2748) 
One list is from ingress nodes (used for protocol classification) and
the second one is from exit nodes (used for flow risk check)
 2025-02-27 10:55:54 +01:00
Luca Deri
050ea7e229 Improved Tor exit node download and added IPv6 support 2025-02-24 19:57:46 +01:00
Luca Deri
501ff5841d Added additional VK ASNs 2025-02-05 15:27:59 +01:00
Ivan Nardi
62d64afde7
Auto-generate Microsoft-related list of domains (#2688) 2025-01-31 15:44:28 +01:00
Fábio Depin
aacade6d95
Force English content in whatsapp_ip_addresses_download.sh (#2701) 
Add the Accept-Language header in the cURL command to ensure the fetched page is in English, improving consistency and clarity in the script output.
 2025-01-30 11:09:24 +01:00
Ivan Nardi
f3532f0bad
Unify "Skype" and "Teams" ids (#2687) 
* Rename `NDPI_PROTOCOL_SKYPE_TEAMS_CALL` ->
  `NDPI_PROTOCOL_MSTEAMS_CALL`

* Rename ip list from "Skype/Teams" to "Teams"
 2025-01-20 18:06:56 +01:00
Luca Deri
511228d36d Added DigitalOcean protocol 2025-01-17 18:26:27 +01:00
Ivan Nardi
6f82e485dc
Update script to download Azure IP list ranges (#2640) 2024-12-11 13:59:31 +01:00
Luca Deri
1c4ae434ab Improved statistics 2024-10-16 23:55:21 +02:00
Luca Deri
97ce729392 Parser for ndpiReader JSON files 2024-10-15 22:25:02 +02:00
Nardi Ivan
5b0374c28b Add detection of SurfShark VPN 2024-09-05 16:36:50 +02:00
Nardi Ivan
f350379e95 Add detection of NordVPN 2024-09-05 16:36:50 +02:00
Ivan Nardi
7558bcd39f
Fix url for downloading X/Twitter crawler IPs (#2526) 2024-08-22 15:35:33 +02:00
Petr
2a3f4dc8b4
Performed some grammar and typo fixes (#2511) 2024-07-19 11:22:35 +02:00
Petr
be0b2c2d90
ipaddr2list.py, ndpi2timeline.py: reformatted (#2509) 2024-07-18 20:35:47 +02:00
Petr
c35a5ca087
shell: reformatted, fixed inspections, typos (#2506) 
Reformatted shell scripts according to [ShellCheck](https://github.com/koalaman/shellcheck/).

I. Most common changes:
1. https://github.com/koalaman/shellcheck/wiki/SC2086
	`$var` → `"$var"`
	Note: this isn't always necessary and I've been careful not to substitute where it wasn't necessary in meaning.
2. https://github.com/koalaman/shellcheck/wiki/SC2006
	`` `command` `` → `$(command)`
3. https://github.com/koalaman/shellcheck/wiki/SC2004
	`$(( $a + $b ))` → `$(( a + b ))`
4. https://github.com/koalaman/shellcheck/wiki/SC2164
	`cd "$dir"` → `cd "$dir" || exit 1`
5. https://github.com/koalaman/shellcheck/wiki/SC2166
	`[ check1 -o check2 ]` → `[ check1 ] || [ check2 ]`
6. https://github.com/koalaman/shellcheck/wiki/SC2002
	`cat "${file}" | wc -c` → `< "${file}" wc -c`
	Note: this looks a bit uglier but works faster.

II. Some special changes:
1. In file `utils/common.sh`:
	https://github.com/koalaman/shellcheck/wiki/SC2112
	This script is interpreted by `sh`, not by `bash`, but uses the keyword `function`.
	So I replaced `#!/usr/bin/env sh` to `#!/usr/bin/env bash`.
2. After that I thought of replacing all shebangs to `#!/usr/bin/env bash` for consistency and cross-platform compatibility, especially since most of the files already use bash.
3. But in cases when it was `#!/bin/sh -e` or `#!/bin/bash -eu` another problem appears:
	https://github.com/koalaman/shellcheck/wiki/SC2096
	So I decided to make all shebangs look uniform:
	```
	#!/usr/bin/env bash
	set -e (or set -eu) (if needed)
	```
4. In file `tests/ossfuzz.sh`:
	https://github.com/koalaman/shellcheck/wiki/SC2162
	`read i` → `read -r i`
	Note: I think that there is no need in special treatment for backslashes, but I could be wrong.
5. In file `tests/do.sh.in`:
	https://github.com/koalaman/shellcheck/wiki/SC2035
	`ls *.*cap*` → `ls -- *.*cap*`
6. In file `utils/verify_dist_tarball.sh`:
	https://github.com/koalaman/shellcheck/wiki/SC2268
	`[ "x${TARBALL}" = x ]` → `[ -z "${TARBALL}" ]`
7. In file `utils/check_symbols.sh`:
	https://github.com/koalaman/shellcheck/wiki/SC2221
	`'[ndpi_utils.o]'|'[ndpi_memory.o]'|'[roaring.o]')` → `'[ndpi_utils.o]'|'[ndpi_memory.o]')`
8. In file `autogen.sh`:
	https://github.com/koalaman/shellcheck/wiki/SC2145
	`echo "./configure $@"` → `echo "./configure $*"`
	https://github.com/koalaman/shellcheck/wiki/SC2068
	`./configure $@` → `./configure "$@"`

III. `LIST6_MERGED` and `LIST_MERGED6`
	There were typos with this variables in files `utils/aws_ip_addresses_download.sh`, `utils/aws_ip_addresses_download.sh` and `utils/microsoft_ip_addresses_download.sh` where variable `LIST6_MERGED` was defined, but `LIST_MERGED6` was removed by `rm`.
	I changed all `LIST_MERGED6` to `LIST6_MERGED`.

Not all changes are absolutely necessary, but some may save you from future bugs.
 2024-07-18 17:32:49 +02:00
Petr
0a3a82680d
python: reformatted, fixed bugs (#2504) 2024-07-17 11:00:42 +02:00
Petr
f8e32bc75b
Fixed mistake in shebang (SC1113) (#2498) 2024-07-15 07:21:03 +02:00
Ivan Nardi
d42f0e6ab3
Add detection of Twitter bot (#2487) 
Update the global list of crawlers ips
 2024-07-03 16:16:54 +02:00
Toni
3639d2045b
Remove unused code. (#2450) 
* some `#ifdef`ed code dates back to 2019, 2020 and 2021
 * some function signatures were still present in `ndpi_main.h`
   which may cause linker errors for libnDPI dependee's
 * return an error while trying to serialize a double instead
   of `fprintf(stderr, ...)`

Signed-off-by: Toni Uhlig <matzeton@googlemail.com>
 2024-05-21 13:08:07 +02:00
Nardi Ivan
478c31b6c9 utils: update script to download Cloudflare ips 2024-02-26 09:26:21 +01:00
Ivan Nardi
12e142565e
Add a script to download/update the domain suffix list (#2321) 2024-02-20 11:51:58 +01:00
Toni
f4d7aa45fe
Improved Polish gambling sites fetch script. (#2315) 
* fails quite often in the CI, so ignore potential xmllint error

Signed-off-by: Toni Uhlig <matzeton@googlemail.com>
 2024-02-10 13:33:56 +01:00
Ivan Nardi
dd8be1fcb1
Fix some warnings reported by CODESonar (#2227) 
Remove some unreached/duplicated code.

Add error checking for `atoi()` calls.

About `isdigit()` and similar functions. The warning reported is:
```
Negative Character Value help
isdigit() is invoked here with an argument of signed type char, but only
has defined behavior for int arguments that are either representable
as unsigned char or equal to the value of macro EOF(-1).
Casting the argument to unsigned char will avoid the undefined behavior.
In a number of libc implementations, isdigit() is implemented using lookup
tables (arrays): passing in a negative value can result in a read underrun.
```
Switching to our macros fix that.
Add a check to `check_symbols.sh` to avoid using the original functions
from libc.
 2024-01-12 13:30:43 +01:00
Toni
5fb631c8fe
Improved belgium gambling sites regex. (#2184) 
Signed-off-by: Toni Uhlig <matzeton@googlemail.com>
 2023-11-29 16:15:23 +01:00
Ivan Nardi
55664392a9
Ip address list: aggregate Mullvad and Tor lists too (#2154) 
Missing from bdb73db1a
See #2150
 2023-11-21 08:28:32 +01:00