Commit graph

21 commits

Author SHA1 Message Date
Ivan Nardi
113170cca4
New protocols for Amazon/AWS sub-classification (#2975)
Add:
* Cognito
* API Gateway
* Kinesis
* EC2
* EMR
* S3
* Cloudfront
* DynamoDB

Keep `NDPI_PROTOCOL_AMAZON_AWS` for generic AWS traffic
2025-10-02 11:48:25 +02:00
JH
ebb47c9a84
Add auto-updating cryptocurrency mining pool lists (#2891)
Co-authored-by: Ivan Nardi <nardi.ivan@gmail.com>
2025-07-01 20:27:22 +02:00
Toni
470a479eb2
Remove ProtonVPN IP address fetch (#2811)
* not available w/o an API token anymore

Signed-off-by: Toni Uhlig <matzeton@googlemail.com>
2025-05-12 13:28:51 +02:00
Ivan Nardi
678d284a25
TOR: fix ip lists (#2748)
One list is from ingress nodes (used for protocol classification) and
the second one is from exit nodes (used for flow risk check)
2025-02-27 10:55:54 +01:00
Ivan Nardi
62d64afde7
Auto-generate Microsoft-related list of domains (#2688) 2025-01-31 15:44:28 +01:00
Luca Deri
511228d36d Added DigitalOcean protocol 2025-01-17 18:26:27 +01:00
Nardi Ivan
5b0374c28b Add detection of SurfShark VPN 2024-09-05 16:36:50 +02:00
Nardi Ivan
f350379e95 Add detection of NordVPN 2024-09-05 16:36:50 +02:00
Petr
c35a5ca087
shell: reformatted, fixed inspections, typos (#2506)
Reformatted shell scripts according to [ShellCheck](https://github.com/koalaman/shellcheck/).

I. Most common changes:
1. https://github.com/koalaman/shellcheck/wiki/SC2086
	`$var` → `"$var"`
	Note: this isn't always necessary and I've been careful not to substitute where it wasn't necessary in meaning.
2. https://github.com/koalaman/shellcheck/wiki/SC2006
	`` `command` `` → `$(command)`
3. https://github.com/koalaman/shellcheck/wiki/SC2004
	`$(( $a + $b ))` → `$(( a + b ))`
4. https://github.com/koalaman/shellcheck/wiki/SC2164
	`cd "$dir"` → `cd "$dir" || exit 1`
5. https://github.com/koalaman/shellcheck/wiki/SC2166
	`[ check1 -o check2 ]` → `[ check1 ] || [ check2 ]`
6. https://github.com/koalaman/shellcheck/wiki/SC2002
	`cat "${file}" | wc -c` → `< "${file}" wc -c`
	Note: this looks a bit uglier but works faster.

II. Some special changes:
1. In file `utils/common.sh`:
	https://github.com/koalaman/shellcheck/wiki/SC2112
	This script is interpreted by `sh`, not by `bash`, but uses the keyword `function`.
	So I replaced `#!/usr/bin/env sh` to `#!/usr/bin/env bash`.
2. After that I thought of replacing all shebangs to `#!/usr/bin/env bash` for consistency and cross-platform compatibility, especially since most of the files already use bash.
3. But in cases when it was `#!/bin/sh -e` or `#!/bin/bash -eu` another problem appears:
	https://github.com/koalaman/shellcheck/wiki/SC2096
	So I decided to make all shebangs look uniform:
	```
	#!/usr/bin/env bash
	set -e (or set -eu) (if needed)
	```
4. In file `tests/ossfuzz.sh`:
	https://github.com/koalaman/shellcheck/wiki/SC2162
	`read i` → `read -r i`
	Note: I think that there is no need in special treatment for backslashes, but I could be wrong.
5. In file `tests/do.sh.in`:
	https://github.com/koalaman/shellcheck/wiki/SC2035
	`ls *.*cap*` → `ls -- *.*cap*`
6. In file `utils/verify_dist_tarball.sh`:
	https://github.com/koalaman/shellcheck/wiki/SC2268
	`[ "x${TARBALL}" = x ]` → `[ -z "${TARBALL}" ]`
7. In file `utils/check_symbols.sh`:
	https://github.com/koalaman/shellcheck/wiki/SC2221
	`'[ndpi_utils.o]'|'[ndpi_memory.o]'|'[roaring.o]')` → `'[ndpi_utils.o]'|'[ndpi_memory.o]')`
8. In file `autogen.sh`:
	https://github.com/koalaman/shellcheck/wiki/SC2145
	`echo "./configure $@"` → `echo "./configure $*"`
	https://github.com/koalaman/shellcheck/wiki/SC2068
	`./configure $@` → `./configure "$@"`

III. `LIST6_MERGED` and `LIST_MERGED6`
	There were typos with this variables in files `utils/aws_ip_addresses_download.sh`, `utils/aws_ip_addresses_download.sh` and `utils/microsoft_ip_addresses_download.sh` where variable `LIST6_MERGED` was defined, but `LIST_MERGED6` was removed by `rm`.
	I changed all `LIST_MERGED6` to `LIST6_MERGED`.

Not all changes are absolutely necessary, but some may save you from future bugs.
2024-07-18 17:32:49 +02:00
Petr
f8e32bc75b
Fixed mistake in shebang (SC1113) (#2498) 2024-07-15 07:21:03 +02:00
Ivan Nardi
12e142565e
Add a script to download/update the domain suffix list (#2321) 2024-02-20 11:51:58 +01:00
Toni
6dcecd73d3
Added malicious sites from the polish cert. (#2121)
* added handling of parsing errors

Signed-off-by: Toni Uhlig <matzeton@googlemail.com>
2023-11-02 09:04:04 +01:00
snicket2100
1fbe8a2385
Mullvad VPN service added (based on entry node IP addresses) (#2062) 2023-08-02 19:44:16 +02:00
Ivan Nardi
3e673e91a9
ProtonVPN: add basic detection (#2006) 2023-06-08 16:52:55 +02:00
Toni
c97e2d67ff
Added scripts to auto generate hostname/SNI *.inc files. (#1984)
* add illegal gambling sites (Belgium)

Signed-off-by: lns <matzeton@googlemail.com>
2023-05-20 15:41:15 +02:00
Luca Deri
bfe79243bc Refreshed lists 2023-05-08 16:51:38 +02:00
Ivan Nardi
6b94c9675a
Improve detection of crawler/bot traffic (#1956) 2023-05-04 11:27:34 +02:00
Toni
b2648a45a3
Added Edgecast and Cachefly CDNs. (#1540)
* Improved ASN update script
 * Ran `utils/update_every_lists.sh'
 * `tests/do.sh.in' prints the amount of failed pcap(s)
 * `utils/asn_update.sh' prints the amount of failed download(s)

Signed-off-by: lns <matzeton@googlemail.com>
2022-05-07 09:33:25 +02:00
Toni
824c343602
Updated `utils/whatsapp_ip_addresses_download.sh' to scrape the required IP addresses/ranges. (#1524)
* Replaced return statements in `utils/*.sh' with exit's (such scripts should never source'd)
 * Ran `utils/update_every_lists.sh'

Signed-off-by: lns <matzeton@googlemail.com>
2022-04-22 13:27:41 +02:00
Toni
f646a4bce0
Improved ASN/IP update scripts and CI integration. (#1474)
* CI will print a warning if ASN/IP addresses changed.

Signed-off-by: Toni Uhlig <matzeton@googlemail.com>
2022-03-09 13:53:04 +01:00
Ivan Nardi
8b062295cc
Add some scripts to easily update some IPs lists (#1449)
While the lists in a6ff0dd0 and 2f5f445f are somehow provided by the
companies themselves (or by some interested parties), these new lists
are directly extracted from BGP information, via AS prefixes.

*Usually*, these new lists are far more stable than the previous ones.

TODO:
* add some other ASNs (see `src/lib/ndpi_content_match.c.inc`)
* IPv6, as usual :-(
2022-02-25 14:26:26 +01:00
Renamed from utils/update_every_content_match_lists.sh (Browse further)