The `suffix_id` is simply an incremental index (see
`ndpi_load_domain_suffixes`), so its value might changes every time we
update the public suffix list.
Export some metadata (for the moment, SNI and TLS fingerprints) to
Wireshark/tshark via extcap.
Note that:
* metadata are exported only once per flow
* metadata are exported (all together) when nDPI stopped processing
the flow
Still room for a lot of improvements!
In particular:
* we need to add some boundary checks (if we are going to export other
attributes)
* we should try to have a variable length trailer
This cache was added in b6b4967aa, when there was no real Zoom support.
With 63f349319, a proper identification of multimedia stream has been
added, making this cache quite useless: any improvements on Zoom
classification should be properly done in Zoom dissector.
Tested for some months with a few 10Gbits links of residential traffic: the
cache pretty much never returned a valid hit.
* Added
size_t ndpi_compress_str(const char * in, size_t len, char * out, size_t bufsize);
size_t ndpi_decompress_str(const char * in, size_t len, char * out, size_t bufsize);
used to compress short strings such as domain names. This code is based on
https://github.com/Ed-von-Schleck/shoco
* Major code rewrite for ndpi_hash and ndpi_domain_classify
* Improvements to make sure custom categories are loaded and enabled
* Fixed string encoding
* Extended SalesForce/Cloudflare domains list
Enable parsing of Mapped-Address attribute for all STUN flows: that
means that STUN classification might require more packets.
Add a configuration knob to enable/disable this feature.
Note that we can have (any) STUN metadata also for flows *not*
classified as STUN (because of DTLS).
Add support for ipv6.
Restore the correct extra dissection logic for Telegram flows.
* Integrated RoaringBitmap v3
* Renamed ndpi_bitmap64 ro ndpi_bitmap64_fuse
* Fixes to ndpi_bitmap for new roaring library
* Fixes for bitmap serialization
* Fixed format
* Warning fix
* Conversion fix
* Warning fix
* Added check for roaring v3 support
* Updated file name
* Updated path
* Uses clang-9 (instead of clang-7) for builds
* Fixed fuzz_ds_bitmap64_fuse
* Fixes nDPI printf handling
* Disabled printf
* Yet another printf fix
* Cleaup
* Fx for compiling on older platforms
* Fixes for old compilers
* Initialization changes
* Added compiler check
* Fixes for old compilers
* Inline function is not static inline
* Added missing include
The basic idea is to have the following logic:
* pattern "DOMAIN" matches the domain itself (i.e exact match) *and* any
subdomains (i.e. "ANYTHING.DOMAIN")
* pattern "DOMAIN." matches *also* any strings for which is a prefix
[please, note that this kind of match is handy but it is quite
dangerous...]
* pattern "-DOMAIN" matches *also* any strings for which is a postfix
Examples:
* pattern "wikipedia.it":
* "wikipiedia.it" -> OK
* "foo.wikipedia.it -> OK
* "foowikipedia.it -> NO MATCH
* "wikipedia.it.com -> NO MATCH
* pattern "wikipedia.":
* "wikipedia.it" -> OK
* "foo.wikipedia.it -> OK
* "foowikipedia.it -> NO MATCH
* "wikipedia.it.com -> OK
* pattern "-wikipedia.it":
* "wikipedia.it" -> NO MATCH
* "foo.wikipedia.it -> NO MATCH
* "0001-wikipedia.it -> OK
* "foo.0001-wikipedia.it -> OK
Bottom line:
* exact match
* prefix with "." (always, implicit)
* prefix with "-" (only if esplicitly set)
* postfix with "." (only if esplicitly set)
That means that the patterns cannot start with '.' anymore.
Close#2330
Add the concept of "global context".
Right now every instance of `struct ndpi_detection_module_struct` (we
will call it "local context" in this description) is completely
independent from each other. This provide optimal performances in
multithreaded environment, where we pin each local context to a thread,
and each thread to a specific CPU core: we don't have any data shared
across the cores.
Each local context has, internally, also some information correlating
**different** flows; something like:
```
if flow1 (PeerA <-> Peer B) is PROTOCOL_X; then
flow2 (PeerC <-> PeerD) will be PROTOCOL_Y
```
To get optimal classification results, both flow1 and flow2 must be
processed by the same local context. This is not an issue at all in the far
most common scenario where there is only one local context, but it might
be impractical in some more complex scenarios.
Create the concept of "global context": multiple local contexts can use
the same global context and share some data (structures) using it.
This way the data correlating multiple flows can be read/write from
different local contexts.
This is an optional feature, disabled by default.
Obviously data structures shared in a global context must be thread safe.
This PR updates the code of the LRU implementation to be, optionally,
thread safe.
Right now, only the LRU caches can be shared; the other main structures
(trees and automas) are basically read-only: there is little sense in
sharing them. Furthermore, these structures don't have any information
correlating multiple flows.
Every LRU cache can be shared, independently from the others, via
`ndpi_set_config(ndpi_struct, NULL, "lru.$CACHE_NAME.scope", "1")`.
It's up to the user to find the right trade-off between performances
(i.e. without shared data) and classification results (i.e. with some
shared data among the local contexts), depending on the specific traffic
patterns and on the algorithms used to balance the flows across the
threads/cores/local contexts.
Add some basic examples of library initialization in
`doc/library_initialization.md`.
This code needs libpthread as external dependency. It shouldn't be a big
issue; however a configure flag has been added to disable global context
support. A new CI job has been added to test it.
TODO: we should need to find a proper way to add some tests on
multithreaded enviroment... not an easy task...
*** API changes ***
If you are not interested in this feature, simply add a NULL parameter to
any `ndpi_init_detection_module()` calls.
Avoid collisions between user-ids and internal-ids protocols in the
`example/protos.txt` file.
Add a new value for the classification confidence:
`NDPI_CONFIDENCE_CUSTOM_RULE`
With `./example/ndpiReader -p example/protos.txt -H` we now see also the
custom protocols and their internal/external ids:
```
nDPI supported protocols:
Id Userd-id Protocol Layer_4 Nw_Proto Breed Category
0 0 Unknown TCP X Unrated Unspecified
...
387 387 Mumble UDP X Fun VoIP
388 388 iSCSI TCP Acceptable Unspecified
389 389 Kibana TCP Acceptable Unspecified
390 390 TestProto TCP Acceptable Unspecified
391 391 HomeRouter TCP Acceptable Unspecified
392 392 CustomProtocol TCP Acceptable Unspecified
393 393 AmazonPrime TCP Acceptable Unspecified
394 394 CustomProtocolA TCP Acceptable Unspecified
395 395 CustomProtocolB TCP Acceptable Unspecified
396 800 CustomProtocolC TCP Acceptable Unspecified
397 1024 CustomProtocolD TCP Acceptable Unspecified
398 2048 CustomProtocolE TCP Acceptable Unspecified
399 2049 CustomProtocolF TCP Acceptable Unspecified
400 2050 CustomProtocolG TCP Acceptable Unspecified
401 65535 CustomProtocolH TCP Acceptable Unspecified
```
We likely need to take a better look in general at the iteration between
internal and external protocols ids...
This PR fixes the issue observed in
https://github.com/ntop/nDPI/pull/2274#discussion_r1460674874 and in
https://github.com/ntop/nDPI/pull/2275.
Some changes in the parameters names.
Add a fuzzer to fuzz the configuration file format.
Add the infrastructure to configuratin callbacks.
Add an helper to map LRU cache indexes to names.
This is the first step into providing (more) configuration options in nDPI.
The idea is to have a simple way to configure (most of) nDPI: only one
function (`ndpi_set_config()`) to set any configuration parameters
(in the present or on in the future) and we try to keep this function
prototype as agnostic as possible.
You can configure the library:
* via API, using `ndpi_set_config()`
* via a configuration file, in a text format
This way, anytime we need to add a new configuration parameter:
* we don't need to add two public functions (a getter and a setter)
* we don't break API/ABI compatibility of the library; even changing
the parameter type (from integer to a list of integer, for example)
doesn't break the compatibility.
The complete list of configuration options is provided in
`doc/configuration_parameters.md`.
As a first example, two configuration knobs are provided:
* the ability to enable/disable the extraction of the sha1 fingerprint of
the TLS certificates.
* the upper limit on the number of packets per flow that will be subject
to inspection
Allow to disable this check.
I don't know how much sense these limits have in the application
(especially with those default values...) since we have always had a
hard limit on the library itself (`max_packets_to_process` set to 32).
The only value might be that they provide different limits for TCP and
UDP traffic.
Keep them for the time being...
