Commit graph

373 commits

Author SHA1 Message Date
Toni
74a77e7b3d
Added --ignore-vlanid / -I to exclude VLAN ids for flow hash calculation. #1073 (#1085)
Signed-off-by: Toni Uhlig <matzeton@googlemail.com>
2020-12-11 21:01:51 +01:00
Luca Deri
eb689b2069 nDPI rules (work in progress) implementation 2020-11-30 22:01:49 +01:00
Luca Deri
48d640583a Moved global in reader_util.c 2020-10-27 08:40:00 +01:00
Luca Deri
948a906037 Added -D flag for detecting DoH in the wild
Removed heuristic from CiscoVPN as it leads to false positives
2020-10-26 21:40:59 +01:00
Luca Deri
9b85669a64 Added -x for checking patterns 2020-10-22 23:28:18 +02:00
Alfredo Cardigliano
e6d206fd15 Add unit tests to travis. Move ndpi serializer tests to unit tests. 2020-09-21 17:24:06 +02:00
Luca Deri
753b5dde16
Merge pull request #1012 from IvanNardi/ua
QUIC: extract User Agent information
2020-09-17 21:32:25 +02:00
Luca Deri
7086197047 Added extension to detect nested subdomains as used in Browsertunnel attack tool
https://github.com/veggiedefender/browsertunnel
2020-09-09 23:25:19 +02:00
Nardi Ivan
a1014e8895 http: create a common function to parse User Agent field
Prepare the code to handle UA information from flows other than HTTP
2020-09-08 10:34:05 +02:00
Toni Uhlig
adab0139c2
Fixed invalid memory access leading to a SIGSEGV in ndpiReader's option parser.
Signed-off-by: Toni Uhlig <matzeton@googlemail.com>
2020-08-28 06:53:58 +02:00
Nardi Ivan
d62ae567d1 Add (optional) dependency on external libraries: libgcrypt and libgpg-error
To support QUIC payload and header decryption, it is necessary to choose an
external crypto library to handle the low-level crypto stuff. Since we will
use some Wireshark code, it is quite natural to choose the same library used
by Wireshark itself: libgcrypt.

More precisely, we will use libgcrypt and libgpg-error.
Both libraries have LGPL license, so there should be no issue from this point
of view.

These libraries are not required to build nDPI, and their usage is optional:
nDPI will keep working (and compiling) even if they are not available.
However, without them, QUIC sub-classification is next to impossible.
The configure flag "--disable-gcrypt" forces the build system to ignore these
libraries.

libgpg-error is only used for debug to have meaningful error messages and its
usage is trivial.
The same cannot be said for libgcrypt because its initialization is a significant
issue.

The rest of this commit message try explaining how libgcrypt is
initialized.

According to the documentation
   https://gnupg.org/documentation/manuals/gcrypt/Initializing-the-library.html
   https://gnupg.org/documentation/manuals/gcrypt/Multi_002dThreading.html#Multi_002dThreading
libgcrypt must be initialized before using it, but such initialization should
be performed by the actual application and not by any library.

Forcing the users to proper initialize libgcrypt in their own code seems
unreasonable: most people using nDPI might be complete unaware of any crypto
stuff and update each and every one application linking to nDPI with specific
libgcrypt code should be out of question, anyway.

Fortunately, it seems a workaround exists to initialize libgcrypt in a library
    https://lists.gnupg.org/pipermail/gcrypt-devel/2003-August/000458.html

Therefore, we could provide a wrapper to this initialization stuff in a nDPI
function. Unfortunately nDPI API lacks a global init function that must be
called only once, before any other functions. We could add it, but that would
be a major API break.
AFAIK, ndpi_init_detection_module() might be called multiple times, for example
to create multiple independent dpi engines in the same program.

The proposed solution is to (optionally) initialize libgcrypt in
ndpi_init_detection_module() anyway:
* if the actual application doesn't directly use libgcrypt and only calls
  ndpi_init_detection_module() once, everything is formally correct and it
  should work out of the box [by far the most common user case];
* if the actual application already uses libgcrypt directly, it already
  performs the required initialization. In this case the ndpi_prefs.ndpi_dont_init_libgcrypt
  flag should be passed to ndpi_init_detection_module() to avoid further
  initializations.

The only scenario not supported by this solution is when the application is
unaware of libgcrypt and calls ndpi_init_detection_module() multiple times
concurrently. But this scenario should be uncommon.

A completely different option should be to switch to another crypto library,
with a huge impact on the QUIC dissector code.

Bottom line: crypto is hard, using libgcrypt is complex and the proposed
initialization, even if not perfect, should cover the most frequent user
cases and should work, for the time being.
If anyone has some suggestions...
2020-08-21 22:04:55 +02:00
Toni Uhlig
13cbd1e4ae
Fixed invalid dpdk fn call.
Signed-off-by: Toni Uhlig <matzeton@googlemail.com>
2020-08-15 12:38:47 +02:00
Luca Deri
d3fb1fb25a Fixed possible memory leak in TLS certificate handling 2020-08-05 17:57:00 +02:00
Luca Deri
a828ac0191 Tiny changes for TLS block lenght dissection 2020-07-29 22:36:27 +02:00
Nardi Ivan
7d843f4d17 Minor fixes
Fix a memory leak and an issue (re)-introduced in configure script
2020-07-26 21:11:34 +02:00
Luca Deri
21e2e57614 Fixed bin similarity 2020-07-24 01:30:58 +02:00
Luca Deri
439558f6a3 Improved bin clustering 2020-07-22 23:56:50 +02:00
Luca Deri
3cd1ec5c9a Added changes for handlign SSSH cipher detection 2020-07-22 15:58:46 +02:00
Luca Deri
802254327a wrapper cleanup 2020-07-13 10:53:25 +02:00
Luca Deri
9d8717a0c7
Merge pull request #963 from yskcg/dev
add improved boundary check and check malloc return is NULL
2020-07-13 10:24:23 +02:00
Toni Uhlig
23c0721538
Fixed race condition in ndpi_ssl_version2str() caused by static qualifier in the version string buffer.
* added also GREASE supported tls versions as specified in
   https://tools.ietf.org/html/draft-davidben-tls-grease-01#page-4

Signed-off-by: Toni Uhlig <matzeton@googlemail.com>
2020-07-11 01:05:39 +02:00
ysk
35f1c362b9 add improved boundary check and check malloc return is NULL 2020-07-10 17:49:35 +08:00
Luca Deri
014fdd9a02 Various fixes in bins implementation
Added -b flag in ndpiReader to test bins
2020-07-09 17:28:02 +02:00
Luca Deri
9d8967236e
Merge pull request #957 from lnslbrty/fix/build
Fix/build
2020-07-09 17:27:53 +02:00
Luca Deri
d658ac9e61 Added -b to ndpiReader to evaluate bins 2020-07-09 00:15:21 +02:00
Toni Uhlig
20fed83e0f
Removed csv_fp as external symbol. Instead passing csv_fp through as argument.
Signed-off-by: Toni Uhlig <matzeton@googlemail.com>
2020-07-08 23:21:35 +02:00
Nardi Ivan
fd6a5bc1cd Fix memory leak reported in #955
This leak is clearly happening because of missing/disabled code in "bins"
handling.
This is a trivial workaround useful until such code is completed.
In this way tavis-ci will stop complaining
2020-07-08 11:30:39 +02:00
Luca Deri
fb04dbbc47 Added testing bin code 2020-07-07 19:18:43 +02:00
Luca Deri
f1dd8bc8c5 Memory allocation fixes 2020-07-07 17:32:26 +02:00
Toni Uhlig
e308e59002
fixed memory leak in ndpi_cluster_bins / binUnitTest
Signed-off-by: Toni Uhlig <matzeton@googlemail.com>
2020-07-07 17:07:20 +02:00
Luca Deri
1c60c22893 Added ndpi_cluster_bins() for clustering bins and ancillary functions for bins manipulation 2020-07-07 15:10:51 +02:00
Luca Deri
db707e0829
Merge pull request #932 from IvanNardi/log
Log
2020-07-07 14:43:32 +02:00
Nardi Ivan
030f3f3d48 Fix a memory leak 2020-07-07 10:27:57 +02:00
Luca Deri
513c857631 Added unit test 2020-07-03 14:43:14 +02:00
Nardi Ivan
974c1cc681 Improve help message of --dbg-proto option
Make it clear that such option is general, not about extcap
functionality
2020-07-02 09:42:53 +02:00
Luca Deri
392ce44573 Added ndpi_print_bin() API call 2020-06-29 22:36:46 +02:00
Luca Deri
2c263bc726 Added ndpi_bin_similarity() for computing bin similarity 2020-06-29 19:09:38 +02:00
Nardi Ivan
b24f5c4c0a Fix memory leak about purged/expired flows
Create an helper to avoid similar errors in the future
Fixes: 1a62f4c7
2020-06-28 12:05:12 +02:00
Nardi Ivan
ece5d3e199 Fix (harmless) memory leaks when DPDK is enabled 2020-06-28 12:05:12 +02:00
Nardi Ivan
2cdf7ce806 Fix startup when DPDK is enabled
Fixes:a58c838c4
2020-06-28 12:05:12 +02:00
Nardi Ivan
2effa57d8a ndpiReader: fix ports statistics
They should take idle/expired flows into account, too
2020-06-28 12:05:12 +02:00
Nardi Ivan
70a926088f Fixed harmless memory leak in extcap initialization 2020-06-25 18:02:03 +02:00
Nardi Ivan
6b69e7ebbc Restore extcap functionality, i.e. integration with wireshark 2020-06-25 18:02:03 +02:00
Toni Uhlig
39800c88fa
Fixed unitialized values in ndpiReader protocol detection bitmask during dga selftest.
* make ./tests/vagrind_test.sh directory agnostic

Signed-off-by: Toni Uhlig <matzeton@googlemail.com>
2020-06-23 17:23:56 +02:00
Luca Deri
d9af1562f0 Fixes #906
Packet bins are not printed wehn empty
2020-06-22 14:30:26 +02:00
Luca Deri
1a62f4c799 Added ndpi_bin_XXX API
Added packet lenght distribution bins
2020-06-22 01:02:54 +02:00
Luca Deri
b2c24558c5 DGA detection improvements 2020-06-18 00:17:30 +02:00
Luca Deri
46d96e7f32 Added checks for DGA detection 2020-06-17 19:46:37 +02:00
Luca Deri
55364ef0b4 Added DGA risk for names that look like a DGA 2020-06-11 18:51:53 +02:00
Luca Deri
60aaa80570 Added HyperLogLog cardinality estimator API calls
/* Memory lifecycle */
int ndpi_hll_init(struct ndpi_hll *hll, u_int8_t bits);
void ndpi_hll_destroy(struct ndpi_hll *hll);

/* Add values */
void ndpi_hll_add(struct ndpi_hll *hll, const char *data, size_t data_len);
void ndpi_hll_add_number(struct ndpi_hll *hll, u_int32_t value) ;

/* Get cardinality estimation */
double ndpi_hll_count(struct ndpi_hll *hll);
2020-06-10 23:43:35 +02:00