Commit graph

77 commits

Author SHA1 Message Date
Toni
65e67a2f88
autoconf: Do not check for json-c if --with-only-libndpi is set. (#1202)
* json-c is used by a unit test
 * required to fix some libnDPI cross compilation issues

Signed-off-by: Toni Uhlig <matzeton@googlemail.com>
2021-06-09 08:01:10 +02:00
Luca Deri
43a8576efb Reworked human readeable string search in flows
Removed fragment manager code
2021-05-17 20:55:06 +02:00
Bernhard M. Wiedemann
0663544be8
Allow to override build date with SOURCE_DATE_EPOCH (#1176)
in order to make builds reproducible.
See https://reproducible-builds.org/ for why this is good
and https://reproducible-builds.org/specs/source-date-epoch/
for the definition of this variable.
2021-05-09 15:09:15 +02:00
Luca Deri
c6a835746e Reworked GeoIP API
int ndpi_get_geoip_asn(struct ndpi_detection_module_struct *ndpi_str,
                        char *ip, u_int32_t *asn);
int ndpi_get_geoip_country_continent(struct ndpi_detection_module_struct *ndpi_str, char *ip,
                        char *country_code, u_int8_t country_code_len,
                        char *continent, u_int8_t continent_len);
2021-04-19 21:39:55 +02:00
Zied Aouini
7f9a18fd05
Make libmaxminddb build configurable. (#1154) 2021-03-14 10:47:28 +01:00
Luca Deri
bce54079d8 Initial geoip support 2021-02-18 22:35:25 +01:00
Luca Deri
be58d78417 (Temporarely) Disabled fragment manager (#1129) 2021-02-05 23:03:08 +01:00
Toni
7255dbe4b7
Added fuzzy targets conditional in tests/do.sh.in which prevents the fuzzer from running if nDPI was configured previously --enable-fuzztargets but not for the current config (may produce invalid results). (#1126)
* fixed possible NULL pointer dereference for memcpy(), src pointer should never be NULL

Signed-off-by: Toni Uhlig <matzeton@googlemail.com>

Co-authored-by: Luca Deri <lucaderi@users.noreply.github.com>
2021-02-04 19:28:52 +01:00
Roberto AGOSTINO
b70ad0e2f1
fragments management added (#1122)
Management of tcp segments managements.

Co-authored-by: ragostino <ragostino73@gmail.com>
Co-authored-by: Luca Deri <lucaderi@users.noreply.github.com>
2021-02-03 10:28:51 +01:00
Toni
399755607d
Disable tests that require libgcrypt if --disable-gcrypt set. (#1121)
Signed-off-by: Toni Uhlig <matzeton@googlemail.com>
2021-01-21 08:59:06 +01:00
Luca Deri
4cb8712a35 Added --with-mipsel for building nDPI on mipsel devices 2020-12-09 11:01:40 +00:00
Luca Deri
b77a4276f9 Fixed JSON-C check 2020-11-30 16:41:58 +01:00
Luca Deri
d4a42ed733 Improved JSON-C detection 2020-11-30 16:30:38 +01:00
Toni
daa1c8ef8c
Fix/overflow and libgerror check (#1068)
* Fixed stack overflow caused by missing buffer space for the trailing \0 added by sprintf()

Signed-off-by: Toni Uhlig <matzeton@googlemail.com>

* Remove the autoconf cache value from the previous and failed check before checking again.

Signed-off-by: Toni Uhlig <matzeton@googlemail.com>
2020-11-26 19:46:31 +01:00
Zied Aouini
3d8fd42307
Implement DGA detection performances tracking workflow. (#1064)
* Implement dga evaluation helper.

* Add test set for DGA classification.

* Add DGA classification performances tracking as part of Travis.

* Add DGA evaluation doc.

* Fix CI on OSX.

* Add missing backquote.
2020-11-16 21:17:16 +01:00
Luca Deri
efbfb7cbd6 Set /usr as installation prefix 2020-10-29 17:45:55 +01:00
Vitaliy Ivanov
10506b5173 Generate NDPI_API_VERSION based on git commands.
https://github.com/ntop/PF_RING/issues/668

Check for .git directory works in case of usual clone of nDPI repo.
But if you integrate is as submodule then .git is a file.

Signed-off-by: Vitaliy Ivanov <vitaliyi@interfacemasters.com>
2020-10-29 12:35:05 +01:00
Toni
822c8e56f7
Improved/autoconf (#1037)
* Switched to PKG_CHECK_MODULES to keep pkg-config checks more portable.

Signed-off-by: Toni Uhlig <matzeton@googlemail.com>

* Improved GCrypt autoconf check to detect a possible gpg-error inter-library dependency.

Signed-off-by: Toni Uhlig <matzeton@googlemail.com>
2020-10-21 15:05:33 +02:00
Luca Deri
8e1a3f5cdc Added JSON-C check for unit tests 2020-09-26 12:50:30 +02:00
Alfredo Cardigliano
e6d206fd15 Add unit tests to travis. Move ndpi serializer tests to unit tests. 2020-09-21 17:24:06 +02:00
Luca Deri
1385f05713
Merge pull request #1017 from lnslbrty/fix/mingw-xcompile
Added support for mingw xcompile.
2020-09-20 18:33:40 +02:00
Toni Uhlig
bcce122665
Fixed mingw build w/o examples/tests/fuzzer.
Signed-off-by: Toni Uhlig <matzeton@googlemail.com>
2020-09-18 19:46:07 +02:00
Luca Deri
5a3956fe27 Added command for specifying the aux files dir 2020-09-08 09:51:02 +02:00
Toni Uhlig
363ab2259c
Fixed broken pkg-config file which did not care about gcrypt/pcre.
Signed-off-by: Toni Uhlig <matzeton@googlemail.com>
2020-08-24 18:12:49 +02:00
Luca Deri
af3ce74e02 Fixes compilation issues introduced by https://github.com/ntop/nDPI/pull/989 2020-08-22 16:41:51 +02:00
Nardi Ivan
d62ae567d1 Add (optional) dependency on external libraries: libgcrypt and libgpg-error
To support QUIC payload and header decryption, it is necessary to choose an
external crypto library to handle the low-level crypto stuff. Since we will
use some Wireshark code, it is quite natural to choose the same library used
by Wireshark itself: libgcrypt.

More precisely, we will use libgcrypt and libgpg-error.
Both libraries have LGPL license, so there should be no issue from this point
of view.

These libraries are not required to build nDPI, and their usage is optional:
nDPI will keep working (and compiling) even if they are not available.
However, without them, QUIC sub-classification is next to impossible.
The configure flag "--disable-gcrypt" forces the build system to ignore these
libraries.

libgpg-error is only used for debug to have meaningful error messages and its
usage is trivial.
The same cannot be said for libgcrypt because its initialization is a significant
issue.

The rest of this commit message try explaining how libgcrypt is
initialized.

According to the documentation
   https://gnupg.org/documentation/manuals/gcrypt/Initializing-the-library.html
   https://gnupg.org/documentation/manuals/gcrypt/Multi_002dThreading.html#Multi_002dThreading
libgcrypt must be initialized before using it, but such initialization should
be performed by the actual application and not by any library.

Forcing the users to proper initialize libgcrypt in their own code seems
unreasonable: most people using nDPI might be complete unaware of any crypto
stuff and update each and every one application linking to nDPI with specific
libgcrypt code should be out of question, anyway.

Fortunately, it seems a workaround exists to initialize libgcrypt in a library
    https://lists.gnupg.org/pipermail/gcrypt-devel/2003-August/000458.html

Therefore, we could provide a wrapper to this initialization stuff in a nDPI
function. Unfortunately nDPI API lacks a global init function that must be
called only once, before any other functions. We could add it, but that would
be a major API break.
AFAIK, ndpi_init_detection_module() might be called multiple times, for example
to create multiple independent dpi engines in the same program.

The proposed solution is to (optionally) initialize libgcrypt in
ndpi_init_detection_module() anyway:
* if the actual application doesn't directly use libgcrypt and only calls
  ndpi_init_detection_module() once, everything is formally correct and it
  should work out of the box [by far the most common user case];
* if the actual application already uses libgcrypt directly, it already
  performs the required initialization. In this case the ndpi_prefs.ndpi_dont_init_libgcrypt
  flag should be passed to ndpi_init_detection_module() to avoid further
  initializations.

The only scenario not supported by this solution is when the application is
unaware of libgcrypt and calls ndpi_init_detection_module() multiple times
concurrently. But this scenario should be uncommon.

A completely different option should be to switch to another crypto library,
with a huge impact on the QUIC dissector code.

Bottom line: crypto is hard, using libgcrypt is complex and the proposed
initialization, even if not perfect, should cover the most frequent user
cases and should work, for the time being.
If anyone has some suggestions...
2020-08-21 22:04:55 +02:00
Luca Deri
59ac73b37f Configure code cleanup 2020-08-19 22:01:03 +02:00
Luca Deri
1f921562d1 Added fix for API versions starting with 0 2020-08-17 18:11:18 +02:00
Luca Deri
34a98abcc0 Added --with-only-libndpi configure option to build just lindpi 2020-08-17 18:00:38 +02:00
Nardi Ivan
7d843f4d17 Minor fixes
Fix a memory leak and an issue (re)-introduced in configure script
2020-07-26 21:11:34 +02:00
Philippe Antoine
b568c26989 Enables fuzz targets even without sanitizer
Should fix ossfuzz build
2020-07-20 10:07:54 +02:00
Toni Uhlig
9bbcc2548e
fix broken sanitizer CFLAGS, --enable-fuzztargets should also set -fsanitize
Signed-off-by: Toni Uhlig <matzeton@googlemail.com>
2020-07-06 22:09:11 +02:00
lucaderi
055422503c configure fixes 2020-07-06 10:38:29 +02:00
Toni Uhlig
c17a21359b
Fixed CodeInspector issues.
* Added compiler search list for AC_PROG_CC, AM_PROG_CC_C_O, AC_PROG_CXX, AC_PROG_CC_STDC
   for Mac OS X only
   The list rendered the AX_CHECK_COMPILE_FLAG([-fsanitize=fuzzer]) useless as it did use
   clang for AX_CHECK_COMPILE_FLAG but gcc during the compile process. Seems broken somehow.

Signed-off-by: Toni Uhlig <matzeton@googlemail.com>
2020-07-05 20:41:02 +02:00
Toni Uhlig
4a6c525db8
Improved fuzz_ndpi_reader which supports now SMP/MT w/o race-coniditions.
./tests/do.sh can supports SMP/MT via environment variables.
Removed -fno-sanitize=shift as well, was fixed by 317d3ffd.

Signed-off-by: Toni Uhlig <matzeton@googlemail.com>
2020-06-29 19:07:59 +02:00
Luca Deri
ff67e09101 Restored compiler check for fuzzy targets 2020-06-27 19:34:57 +02:00
Luca Deri
76e6c38459 Use clang where available and fallback to gcc when missing 2020-06-27 19:15:52 +02:00
lucaderi
3034e53d00 CentOS6 fix (santize won't work as too old system)
Fixes warning
2020-06-25 10:22:16 +02:00
Toni Uhlig
678fbcdfd6
Allow -fsanitize for LLVM as well as for GCC.
* renamed --with-llvm-sanitizer to --with-sanitizer
 * disable all optimisations (-O0) if --with-sanitizer set,
   no functions/paramaters/variables will be optimised, improves debugging
 * enable undefined behaviour sanitizer (ubsan)
 * enable leak sanitizer (lsan)
 * check if -fsanitize=fuzzer is available and --enable-fuzztargets set,
   fail if not (only supported by clang)
 * include level 3 debugging information (-g3), improves macro debugging
 * disabled ubsan misaligned pointer access and lshift overflow

Signed-off-by: Toni Uhlig <matzeton@googlemail.com>
2020-06-23 00:22:33 +02:00
Luca Deri
2ccd2c204b API cleanup for indetifying explicitly in automa's what we're searching (protocol or category)
Removed hyperscan support that is apperently unused
2020-05-06 23:19:59 +02:00
Luca Deri
65b428899d Fix for nDPI code not downlooaded from git 2020-04-22 16:15:15 +02:00
emanuele-f
1b8f7ed101 Fix API version generation 2020-03-25 14:07:19 +01:00
Luca Deri
47a4507d9e NDPI_API_VERSION is now computed automatically 2020-03-25 12:51:06 +01:00
emanuele-f
428661c00b Add debug symbols when using --with-llvm-sanitizer 2020-02-14 11:09:00 +01:00
emanuele-f
5c52c00ea4 Add the --with-llvm-sanitizer flag
To enable the leaks detector, set the ASAN_OPTIONS=detect_leaks=1
environment variable, e.g:

	ASAN_OPTIONS=detect_leaks=1 example/ndpiReader -i test.pcap
2020-02-13 18:42:13 +01:00
Luca
6dbe81588f Added --with-pcre to enable libpcre usage 2020-02-02 22:07:21 +01:00
lucaderi
39b1ad3087 PCRE fix 2020-01-31 07:09:54 +01:00
Luca Deri
c68ea3067b Added libpcre lib if present 2020-01-30 22:24:24 +01:00
Luca Deri
c84ed5a712 Added check for libpcre presence 2020-01-29 00:41:35 +01:00
Luca Deri
d1fb41a161 Minor cleanup 2020-01-05 18:42:36 +01:00