Commit graph

1122 commits

Author SHA1 Message Date
Ivan Nardi
fbb7b8c578 custom rules: try to have a coherent behaviour
Custom rules with *new* protocols are checked "first": if there is a
match, the first packet of the flow provides a complete and final
classification.

The same logic should apply to custom rules with "existing" protocols:
if there is match, nDPI shouldn't do anything else.

Remove the `tcp:3000@ntop` custom rule.

Fix the default port for ElasticSearch (in the protocol file)
2025-03-04 18:15:00 +01:00
Ivan Nardi
1439abe23c ndpiReader: remove redundant checks 2025-02-28 19:11:59 +01:00
Ivan Nardi
2d3f08362e
RTP: payload type info should be set only for real RTP flows (#2742) 2025-02-22 13:35:40 +01:00
Ivan Nardi
cde5773762
Move rtp info out of flow->protos (#2739)
Thiw way, the code is ready to handle rtp info from STUN flows too.
And, most important, this change works as workaround to fix some crashes
reported by oss-fuzz
2025-02-21 19:17:28 +01:00
Luca Deri
30c3613f2f Improved RTP dissection with EVS and other mobile voice codecs 2025-02-20 22:59:23 +01:00
Luca Deri
3bef87666d Exported RTP payload in packet metadata
Added ndpi_rtp_payload_type2str() API call
2025-02-19 11:34:19 +01:00
Luca Deri
e052e5b6b7 Further domain checks 2025-02-19 10:32:30 +01:00
Ivan Nardi
5f8545d97a
SSDP: add configuration for disabling metadata extraction (#2736) 2025-02-17 15:16:37 +01:00
Ivan Nardi
baca06bfd2
ndpiReader: print more DNS information (#2717) 2025-02-11 18:16:55 +01:00
Ivan Nardi
1dccaf37b0
DNS: fix check for DGA domain (#2716)
If we have a (potential) valid sub-classification, we shoudn't check for
DGA, even if the subclassification itself is disabled!
2025-02-11 15:48:53 +01:00
Luca Deri
1577955fca Added ndpi_find_protocol_qoe() API call
Updated (C)
2025-02-10 21:21:51 +01:00
Ivan Nardi
dd4807f8ee
bittorrent: add configuration for "hash" metadata (#2706)
Fix confidence value for same TCP flows
2025-01-31 17:42:47 +01:00
Ivan Nardi
cf8f761b93
HTTP: add configuration for some metadata (#2704)
Extend file configuration for just subclassification.
2025-01-31 16:26:53 +01:00
Ivan Nardi
ecf0f8ace3
Create a specific configuration for classification only (#2689)
In some scenarios, you might not be interested in flow metadata or
flow-risks at all, but you might want only flow (sub-)classification.
Examples: you only want to forward the traffic according to the
classification or you are only interested in some protocol statistics.

Create a new configuration file (for `ndpiReader`, but you can trivially
adapt it for the library itself) allowing exactly that. You can use it
via: `ndpiReader --conf=example/only_classification.conf ...`

Note that this way, the nDPI overhead is lower because it might need
less packets per flow:
* TLS: nDPI processes only the CH (in most cases) and not also the SH
  and certificates
* DNS: only the request is processed (instead of both request and
  response)

We might extend the same "shortcut-logic" (stop processing the flow
immediately when there is a final sub-classification) for others
protocols.

Add the configuration options to enable/disable the extraction of some
TLS metadata.
2025-01-31 15:10:30 +01:00
Ivan Nardi
d4fb7b0aa1
fuzz: extend fuzzing coverage (#2696) 2025-01-23 15:23:01 +01:00
Ivan Nardi
9a5533d796
ndpiReader: add some global statistics about FPC (#2680)
Enabled via `--dump-fpc-stats` option
2025-01-17 11:47:37 +01:00
Ivan Nardi
af011e338e
TLS: remove JA3C (#2679)
Last step of removing JA3C fingerprint

Remove some duplicate tests: testing with ja4c/ja3s disabled is already
performed by `disable_metadata_and_flowrisks` configuration.

Close:#2551
2025-01-14 15:02:20 +01:00
Ivan Nardi
63a3547f99
Add (kind of) support for loading a list of JA4C malicious fingerprints (#2678)
It might be usefull to be able to match traffic against a list of
suspicious JA4C fingerprints

Use the same code/logic/infrastructure used for JA3C (note that we are
going to remove JA3C...)

See: #2551
2025-01-14 12:05:03 +01:00
Luca Deri
6b686be9d3
Fix code scanning alert no. 7: Multiplication result converted to larger type (#2677)
Co-authored-by: Copilot Autofix powered by AI <62310815+github-advanced-security[bot]@users.noreply.github.com>
2025-01-13 20:13:46 +01:00
Luca Deri
1a1fa63dda Fixes https://github.com/ntop/nDPI/issues/2672 2025-01-13 20:06:31 +01:00
Ivan Nardi
72fd940301
Remove JA3C output from ndpiReader (#2667)
Removing JA3C is an big task. Let's start with a simple change having an
huge impact on unit tests: remove printing of JA3C information from
ndpiReader.

This way, when we will delete the actual code, the unit tests diffs
should be a lot simpler to look at.

Note that the information if the client/server cipher is weak or
obsolete is still available via flow risk

See: #2551
2025-01-12 13:24:27 +01:00
Ivan Nardi
c3d19be26f
ndpiReader: update JA statistics (#2646)
Show JA4C and JA3S information (instead of JA3C and JA3S)
See #2551 for context
2025-01-06 15:09:25 +01:00
Ivan Nardi
2e20f670dd
QUIC: extract "max idle timeout" parameter (#2649)
Even if it is only the proposed value by the client (and not the
negotiated one), it might be use as hint for timeout by the (external)
flows manager
2025-01-06 13:45:12 +01:00
Ivan Nardi
cae9fb9989
TLS: remove ESNI support (#2648)
ESNI has been superseded by ECH for years, now.
See: https://blog.cloudflare.com/encrypted-client-hello/
Set the existing flow risk if we still found this extension.
2025-01-06 11:04:50 +01:00
Ivan Nardi
803410542e
STUN/RTP: improve metadata extraction (#2641) 2024-12-11 15:28:00 +01:00
Ivan Nardi
a156d69ea4
STUN: fix monitoring (#2639) 2024-12-06 20:19:28 +01:00
Luca Deri
b1d1f8cfe0 STUN counter changes 2024-11-29 17:13:43 +01:00
Luca Deri
6b6b5c7c4e Enhanced STUN stats 2024-11-28 23:19:25 +01:00
Ivan Nardi
4974f806f9
Add a configuration file to ndpiReader (#2629)
Example:
./example/ndpiReader --conf=./example/calls.conf -i ./tests/pcap/signal_videocall.pcapng -v2

Close #2608
2024-11-27 08:24:31 +01:00
Ivan Nardi
cff8bd1bb2
Update flow->flow_multimedia_types to a bitmask (#2625)
In the same flow, we can have multiple multimedia types
2024-11-25 10:12:48 +01:00
Alfredo Cardigliano
2dbf6b3cf8 Remove unused variable 2024-11-18 16:55:50 +01:00
Ivan Nardi
d7d942586a
ndpiReader: fix statistic about total number of flows (#2622)
When capturing live traffic, accounting and export of expired flows is
 wrong (see #2617).
Let's try to fix some statistics, at least
2024-11-18 13:01:53 +01:00
Ivan Nardi
1bda2bf414 SIP: extract some basic metadata 2024-11-12 13:34:25 +01:00
Toni
ccbbcdf283
Unify ndpi debug logging to always use a u16 protocol id (#2613)
* fixes SonarCloud complaint

Signed-off-by: Toni Uhlig <matzeton@googlemail.com>
2024-11-11 15:17:33 +01:00
Luca Deri
35ef56cc24 Crash fix when -f is specified with a non-existing pcap file (-i) 2024-11-04 22:14:09 +01:00
Luca Deri
183175fc6b Exported is_ndpi_proto definition 2024-11-04 22:02:13 +01:00
Luca Deri
412ca8700f Added HTTP credentials extraction 2024-10-31 21:20:46 +01:00
Ivan Nardi
9da99075aa
TLS: export heuristic fingerprint as metadata (#2609) 2024-10-28 23:36:51 +01:00
Ivan Nardi
a55d8d95bc
ndpiReader: fix command line options used by wireshark (#2605) 2024-10-21 18:26:28 +02:00
Ivan Nardi
9021e08901
ndpiReader: explicitly remove non ipv4/6 packets (#2601) 2024-10-19 21:44:32 +02:00
Ivan Nardi
51556113a3
ndpiReader: add some statistics about monitoring (#2602) 2024-10-19 19:44:00 +02:00
Luca Deri
0cc84e4fdd Improved TCP fingepring calculation
Adde basidc OS detection based on TCP fingerprint
2024-10-18 23:47:34 +02:00
Luca Deri
afc4d9e34d Added -L <domain suffix> for loading domain suffixes
Exported domainanme in JSON file (-K JSON)
2024-10-15 22:25:48 +02:00
Luca Deri
6b6dad4fdb Implemented nDPI TCP fingerprint 2024-10-15 21:21:05 +02:00
Ivan Nardi
521d0ca7a0
Add monitoring capability (#2588)
Allow nDPI to process the entire flows and not only the first N packets.
Usefull when the application is interested in some metadata spanning the
entire life of the session.

As initial step, only STUN flows can be put in monitoring.

See `doc/monitoring.md` for further details.

This feature is disabled by default.

Close #2583
2024-10-14 18:05:35 +02:00
Luca Deri
faaa5c5799 Added support for printing JA4r when enabled 2024-10-11 19:24:06 +02:00
Ivan Nardi
3e74c95d19
Fix unit tests on CI on GitHub Actions (#2587)
On CI, tests run in parallel, because of `NDPI_FORCE_PARALLEL_UTESTS`
define
2024-10-11 13:08:48 +02:00
Luca Deri
2e5edd2cc9 Added -N option for dumping/restoring the DNS cache (when enabled)
Example ndpiReader -i en0 --cfg=dpi.address_cache_size,32768 -N /tmp/a
2024-10-10 18:10:03 +02:00
Luca Deri
b9348e9d6e Added new API calls for serializing/restoring the DNS cache
-  bool ndpi_address_cache_dump(struct ndpi_address_cache *cache, char *path, u_int32_t epoch_now);
-  u_int32_t ndpi_address_cache_restore(struct ndpi_address_cache *cache, char *path, u_int32_t epoch_now);
2024-10-10 16:51:45 +02:00
Luca Deri
89316ab4c0 Test unit fix 2024-10-08 08:58:24 +02:00