* fixed possible memory leak caused by an invalid call to `node_proto_guess_walker()` during serialization
* execute serialization code while running regression tests
Signed-off-by: Toni Uhlig <matzeton@googlemail.com>
* Add some statistics to ndpiReader
The purpose of this version of ndpiReader is too adding some other statistics printed by ndpiReader. In this simple version the domain names(in the flows) that are collected are:
flow-> ssh_tls.server_info
flow-> host_server_name
and are placed in a UT_hash_table, ordering them by number of occurrences.
* Update example/ndpiReader.c
Co-authored-by: Toni <matzeton@googlemail.com>
* Update example/ndpiReader.c
Co-authored-by: Toni <matzeton@googlemail.com>
* Update example/ndpiReader.c
Co-authored-by: Toni <matzeton@googlemail.com>
* Update example/ndpiReader.c
Co-authored-by: Toni <matzeton@googlemail.com>
* Update example/ndpiReader.c
Co-authored-by: Toni <matzeton@googlemail.com>
* Update example/ndpiReader.c
Co-authored-by: Toni <matzeton@googlemail.com>
* Update example/ndpiReader.c
Co-authored-by: Toni <matzeton@googlemail.com>
* Update example/ndpiReader.c
Co-authored-by: Toni <matzeton@googlemail.com>
* Update example/ndpiReader.c
Co-authored-by: Toni <matzeton@googlemail.com>
* Update example/ndpiReader.c
Co-authored-by: Toni <matzeton@googlemail.com>
* Update example/ndpiReader.c
Co-authored-by: Toni <matzeton@googlemail.com>
* Update example/ndpiReader.c
Co-authored-by: Toni <matzeton@googlemail.com>
* Update ndpiReader.c
* Update ndpiReader.c
* Update ndpiReader.c
* Update ndpiReader.c
Co-authored-by: Toni <matzeton@googlemail.com>
* Fixes#1528
* Serialization Interface should also fuzzed
* libjson-c may only be used in the unit test to verify the internal serialization interface
* Serialization Interface supports tlv(broken), csv and json
* Unit test does work again and requires libjson-c
Signed-off-by: lns <matzeton@googlemail.com>
* Removed Visual Studio leftovers. Maintaining an autotools project with VS integration requires some additional overhead.
Signed-off-by: Toni Uhlig <matzeton@googlemail.com>
Signed-off-by: lns <matzeton@googlemail.com>
- partial revert of:
commit 51cfdfb0d8
Author: Luca Deri <deri@ntop.org>
Date: Sun Nov 17 17:51:45 2019 +0100
Removed unused JSON-C code
- Json option is changed from 'j' to 'k' as it's used in the new codebase.
- use HAVE_LIBJSON_C instead of HAVE_JSON_C.
- tabs vs spaces clean ups.
Signed-off-by: Vitaliy Ivanov <vitaliyi@interfacemasters.com>
Conflicts:
example/ndpiReader.c
At every fuzz iteration (i.e for every trace file):
* keep the same ndpi context (`ndpi_init_detection_module` is very
slow);
* reset the flow table, otherwise it grows indefinitely.
This change should fix the "out-of-memory" errors reported by oss-fuzz.
Remove the last uses of `struct ndpi_id_struct`.
That code is not really used and it has not been updated for a very long
time: see #1279 for details.
Correlation among flows is achieved via LRU caches.
This change allows to further reduce memory consumption (see also
91bb77a8).
At nDPI 4.0 (more precisly, at a6b10cf, because memory stats
were wrong until that commit):
```
nDPI Memory statistics:
nDPI Memory (once): 221.15 KB
Flow Memory (per flow): 2.94 KB
```
Now:
```
nDPI Memory statistics:
nDPI Memory (once): 235.27 KB
Flow Memory (per flow): 688 B <--------
```
i.e. memory usage per flow has been reduced by 77%.
Close#1279
This allows to avoid triggering alerts for trusted albeit private certificate issuers.
Extended the example/protos.txt with the new syntax for specifying trusted issueDN.
Example:
trusted_issuer_dn:"CN=813845657003339838, O=Code42, OU=TEST, ST=MN, C=US"
As a general rule, the higher the confidence value, the higher the
"reliability/precision" of the classification.
In other words, this new field provides an hint about "how" the flow
classification has been obtained.
For example, the application may want to ignore classification "by-port"
(they are not real DPI classifications, after all) or give a second
glance at flows classified via LRU caches (because of false positives).
Setting only one value for the confidence field is a bit tricky: more
work is probably needed in the next future to tweak/fix/improve the logic.
Remove some unused fields and re-organize other ones.
In particular:
* Update the parameters of `ndpi_ssl_version2str()` function
* Zattoo, Thunder: these timestamps aren't really used.
* Ftp/mail: these protocols are dissected only over TCP.
* Attention must be paid to TLS.Bittorrent flows to avoid invalid
read/write to `flow->protos.bittorrent.hash` field.
This is the last(?) commit of a long series (see 22241a1d, 227e586e,
730c2360, a8ffcd8b) aiming to reduce library memory consumption.
Before, at nDPI 4.0 (more precisly, at a6b10cf7, because memory stats
were wrong until that commit):
```
nDPI Memory statistics:
nDPI Memory (once): 221.15 KB
Flow Memory (per flow): 2.94 KB
```
Now:
```
nDPI Memory statistics:
nDPI Memory (once): 231.71 KB
Flow Memory (per flow): 1008 B <---------
```
i.e. memory usage per flow has been reduced by 66%, dropping below the
psychological threshold of 1 KB.
To further reduce this value, we probably need to look into #1279:
let's fight this battle another day.
Looking at `struct ndpi_flow_struct` the two bigger fields are
`host_server_name[240]` (mainly for HTTP hostnames and DNS domains) and
`protos.tls_quic.client_requested_server_name[256]`
(for TLS/QUIC SNIs).
This commit aims to reduce `struct ndpi_flow_struct` size, according to
two simple observations:
1) maximum one of these two fields is used for each flow. So it seems safe
to merge them;
2) even if hostnames/SNIs might be very long, in practice they are rarely
longer than a fews tens of bytes. So, using a (single) large buffer is a
waste of memory for all kinds of flows. If we need to truncate the name,
we keep the *last* characters, easing domain matching.
Analyzing some real traffic, it seems safe to assume that the vast
majority of hostnames/SNIs is shorter than 80 bytes.
Hostnames/SNIs are always converted to lowercase.
Attention was given so as to be sure that unit-tests outputs are not
affected by this change.
Because of a bug, TLS/QUIC SNI were always truncated to 64 bytes (the
*first* 64 ones): as a consequence, there were some "Suspicious DGA
domain name" and "TLS Certificate Mismatch" false positives.
We can write to `flow->protos` only after a proper classification.
This issue has been found in Kerberos, DHCP, HTTP, STUN, IMO, FTP,
SMTP, IMAP and POP code.
There are two kinds of fixes:
* write to `flow->protos` only if a final protocol has been detected
* move protocol state out of `flow->protos`
The hard part is to find, for each protocol, the right tradeoff between
memory usage and code complexity.
Handle Kerberos like DNS: if we find a request, we set the protocol
and an extra callback to further parsing the reply.
For all the other protocols, move the state out of `flow->protos`. This
is an issue only for the FTP/MAIL stuff.
Add DHCP Class Identification value to the output of ndpiReader and to
the Jason serialization.
Extend code coverage of fuzz tests.
Close#1343Close#1342
`ndpiReader` is only an example, aiming to show nDPI capabilities
and integration, without any claim about performances.
Nonetheless its memory usage per flow is *huge*, limiting the kinds
of traces that we can test on a "normal" hardware (example: scan
attacks).
The key reason of that behaviour is that we preallocate all the memory
needed for *all* the available features.
Try to reduce memory usage simply allocating some structures only
when they are really needed. Most significant example: JOY algorithms.
This way we should use a lot less memory in the two most common
user-cases:
* `ndpiReader` invoked without any particular flag (i.e `ndpiReader -i
$FILENAME_OR_IFACE`)
* internal unit tests
Before (on x86_64):
```
struct ndpi_flow_info {
[...]
/* size: 7320, cachelines: 115, members: 72 */
```
After:
```
struct ndpi_flow_info {
[...]
/* size: 2128, cachelines: 34, members: 75 */
```
This protocol is detected via HTTP Content-Type header.
Until 89d548f9, nDPI had a dedicated automa (`content_automa`) to
classify a HTTP flow according to this header. Since then, this automa has
been useless because it is always empty.
Re-enable it to match only a string seems overkilling.
Remove all `content_automa` leftovers.
Fix all the warnings.
Getting rid of "-Wno-unused-parameter" is quite complex because some
parameters usage depends on compilation variable (i.e.
`--enable-debug-messages`).
The "-Werror" flag has been added only in Travis builds to avoid
breaking the builds to users using uncommon/untested
OS/compiler/enviroment.
Tested on:
* x86_64; Ubuntu 20.04; gcc 7,8,9,10,11; clang 7,8,9,10,11,12
* x86_64; CentOS 7.7; gcc 4.8.5 (with "--disable-gcrypt" flag)
* Raspberry 4; Debian 10.10; gcc 8.3.0
