Commit graph

30 commits

Author SHA1 Message Date
Nardi Ivan
c1b6aa9483 Sync unit tests results 2022-10-19 13:01:34 +02:00
Toni
01622c4c63
Updated protocol dissector serialisation code. (#1767)
Signed-off-by: Toni Uhlig <matzeton@googlemail.com>

Signed-off-by: Toni Uhlig <matzeton@googlemail.com>
2022-10-15 12:58:44 +02:00
Toni
c83e0b3c8d
Restore Confidence enum constants to ascending values. (#1760)
* NDPI_CONFIDENCE_MAX is useless otherwise since it is
   supposed the be the highest value w/o any gaps
 * for client applications, it makes sense to use that
   the numbers directly w/o any additional mapping

Signed-off-by: Toni Uhlig <matzeton@googlemail.com>

Signed-off-by: Toni Uhlig <matzeton@googlemail.com>
2022-10-02 12:47:56 +02:00
Ivan Nardi
a7c2734b38
Remove classification "by-ip" from protocol stack (#1743)
Basically:
* "classification by-ip" (i.e. `flow->guessed_protocol_id_by_ip` is
NEVER returned in the protocol stack (i.e.
`flow->detected_protocol_stack[]`);
* if the application is interested into such information, it can access
`ndpi_protocol->protocol_by_ip` itself.

There are mainly 4 points in the code that set the "classification
by-ip" in the protocol stack:  the generic `ndpi_set_detected_protocol()`/
`ndpi_detection_giveup()` functions and the HTTP/STUN  dissectors.

In the unit tests output, a print about `ndpi_protocol->protocol_by_ip`
has been added for each flow: the huge diff of this commit is mainly due
to that.

Strictly speaking, this change is NOT an API/ABI breakage, but there are
important differences in the classification results. For examples:
* TLS flows without the initial handshake (or without a matching
SNI/certificate) are simply classified as `TLS`;
* similar for HTTP or QUIC flows;
* DNS flows without a matching request domain are simply classified as
`DNS`; we don't have `DNS/Google` anymore just because the server is
8.8.8.8 (that was an outrageous behaviour...);
* flows previusoly classified only "by-ip" are now classified as
`NDPI_PROTOCOL_UNKNOWN`.

See #1425 for other examples of why adding the "classification by-ip" in
the protocol stack is a bad idea.

Please, note that IPV6 is not supported :(  (long standing issue in nDPI) i.e.
`ndpi_protocol->protocol_by_ip` wil be always `NDPI_PROTOCOL_UNKNOWN` for
IPv6 flows.

Define `NDPI_CONFIDENCE_MATCH_BY_IP` has been removed.

Close #1687
2022-09-20 22:24:47 +02:00
Alfredo Cardigliano
973950d881 Replace obsolete linux macro 2022-09-13 10:41:44 +02:00
Toni
37f918322c
Serialize nDPI detection/dissection data as object. (#1725)
Signed-off-by: Toni Uhlig <matzeton@googlemail.com>

Signed-off-by: Toni Uhlig <matzeton@googlemail.com>
2022-09-06 11:38:31 +02:00
Alfredo Cardigliano
bd29fa46ba Serializer supports double for JSON and CSV only for the time being (TLV needs to be extended) 2022-09-01 14:43:52 +02:00
lns
93d65ed650 Support serialization of double-precision floating-point numbers. Fixes #1702.
Signed-off-by: lns <matzeton@googlemail.com>
Signed-off-by: Toni Uhlig <matzeton@googlemail.com>
2022-08-24 10:49:27 +02:00
Toni
b3e722e5a8
Improved nDPI JSON serialization. (#1689)
* fixed autoconf CFLAGS/LDFLAGS MSAN issue which could lead to build errors
 * introduced portable version of gmtime_r aka ndpi_gmtime_r
 * do as most as possible of the serialization work in ndpi_utils.c
 * use flow2json in ndpiReader

Signed-off-by: lns <matzeton@googlemail.com>
2022-08-02 17:54:44 +02:00
Toni
7c19de4904
Do not interfere with CFLAGS/LDFLAGS env anymore. (#1659)
* CI fixes
 * some build systems do not like that (e.g. OpenWrt)
 * fixed some rrdtool related build warnings/errors

Signed-off-by: Toni Uhlig <matzeton@googlemail.com>
2022-07-13 19:44:18 +02:00
Ivan Nardi
5aa3d9126f
Add two new confidence values: confidence by partial DPI (#1632)
Used for all classifications based on partial/incomplete DPI
information, i.e. all classifications done in `ndpi_detection_giveup()`.
2022-07-04 13:56:51 +02:00
Toni
df0ff9bcbd
Added gprof CPU/HEAP profiling support. (#1592)
* Some small auto{conf,make} improvements

Signed-off-by: lns <matzeton@googlemail.com>
2022-06-12 21:00:41 +02:00
Toni
00ee364a7e
Updated README.md (#1562)
* make check great again (not so much)
 * make doc/doc-view
 * CI updates

Signed-off-by: lns <matzeton@googlemail.com>
2022-05-30 19:48:05 +02:00
Toni
87f93ea4fd
Replaced ndpiReader's libjson-c support with libnDPI's internal serialization interface. (#1535)
* Fixes #1528
 * Serialization Interface should also fuzzed
 * libjson-c may only be used in the unit test to verify the internal serialization interface
 * Serialization Interface supports tlv(broken), csv and json
 * Unit test does work again and requires libjson-c

Signed-off-by: lns <matzeton@googlemail.com>
2022-05-07 09:26:09 +02:00
Toni Uhlig
c3df3a12aa Fixed msys2 build warnings and re-activated CI Mingw64 build.
* Removed Visual Studio leftovers. Maintaining an autotools project with VS integration requires some additional overhead.

Signed-off-by: Toni Uhlig <matzeton@googlemail.com>
Signed-off-by: lns <matzeton@googlemail.com>
2022-04-14 19:17:48 +02:00
Toni
f14dcb8344
Provide some API functions for convenience. (#1456)
* Extended JSON serializsation: risk, risk score, confidence

Signed-off-by: Toni Uhlig <matzeton@googlemail.com>
2022-02-25 14:26:04 +01:00
Alfredo Cardigliano
a0bb0ab0ca Add unit test for ndpi_serialize_string_int64 2022-01-21 13:05:58 +01:00
Sam James
c2b7d77784
build: respect environment options more (#1392)
* build: update m4/ax_pthread.m4 from serial 23 -> serial 31

Update ax_pthread.m4 to the latest version from the autoconf-archive
project.

Signed-off-by: Sam James <sam@gentoo.org>

* build: properly detect AR, CC, RANLIB

It's necessary to be able to override choice of AR/CC/RANLIB and other toolchain
variables/tools for cross-compilation, testing with other toolchains, and
to ensure the compiler chosen by the user is actually used for the build.

Previously, GNU_PREFIX was kind-of used for this but this isn't a standard
variable (at all) and it wasn't applied consistently anyway.

We now use the standard autoconf mechanisms for finding these tools.

(RANLIB is already covered by LT_INIT.)

Signed-off-by: Sam James <sam@gentoo.org>

* build: use $(MAKE)

This ensures that parallel make works correctly, as otherwise, a fresh
make job will be started without the jobserver fd, and hence
not know about its parent, forcing -j1.

* build: respect CPPFLAGS, LDFLAGS

- CPPFLAGS is for the C preprocessor (usually for setting defines)
- LDFLAGS should be placed before objects for certain flags to work
  (e.g. -Wl,--as-needed)

Signed-off-by: Sam James <sam@gentoo.org>

Co-authored-by: Luca Deri <lucaderi@users.noreply.github.com>
2022-01-18 14:30:14 +01:00
Luca Deri
5e0befd56c Fixed seriealizer unit 2021-11-30 17:24:42 +01:00
Ivan Nardi
8fdffbf3a1
Compile everything with "-W -Wall -Wno-unused-parameter" flags (#1276)
Fix all the warnings.

Getting rid of "-Wno-unused-parameter" is quite complex because some
parameters usage depends on compilation variable (i.e.
`--enable-debug-messages`).

The "-Werror" flag has been added only in Travis builds to avoid
breaking the builds to users using uncommon/untested
OS/compiler/enviroment.

Tested on:
* x86_64; Ubuntu 20.04; gcc 7,8,9,10,11; clang 7,8,9,10,11,12
* x86_64; CentOS 7.7; gcc 4.8.5 (with "--disable-gcrypt" flag)
* Raspberry 4; Debian 10.10; gcc 8.3.0
2021-08-20 18:11:13 +02:00
Toni
8d0c7b1fae
Fixed Mingw64 build, SonerCloud-CI and more. (#1273)
* Added ARM build and unit test run for SonarCloud-CI.

Signed-off-by: Toni Uhlig <matzeton@googlemail.com>

* Fixed Mingw64 build.

 * adapted to SonarCloud-CI workflow
 * removed broken and incomplete Windows example (tested on VS2017/VS2019)
 * removed unnecessary include (e.g. pthread.h for the library which does not make use of it)

Signed-off-by: Toni Uhlig <matzeton@googlemail.com>
2021-08-18 11:34:16 +02:00
Toni
13c5d6801e
Improved libm check via Autoconf. (#1263)
* libm will now be part of @ADDITIONAL_LIBS@ (if required)

Signed-off-by: Toni Uhlig <matzeton@googlemail.com>
2021-07-28 22:24:03 +02:00
Toni
822c8e56f7
Improved/autoconf (#1037)
* Switched to PKG_CHECK_MODULES to keep pkg-config checks more portable.

Signed-off-by: Toni Uhlig <matzeton@googlemail.com>

* Improved GCrypt autoconf check to detect a possible gpg-error inter-library dependency.

Signed-off-by: Toni Uhlig <matzeton@googlemail.com>
2020-10-21 15:05:33 +02:00
Toni
c2d8955c18
Fixed missing PCAP include directories in Makefiles. (#1034)
Signed-off-by: Toni Uhlig <matzeton@googlemail.com>
2020-10-19 11:25:22 +02:00
Luca Deri
b68a3707f6 Updated serialization test unit 2020-10-02 12:39:08 +02:00
Luca Deri
8e1a3f5cdc Added JSON-C check for unit tests 2020-09-26 12:50:30 +02:00
lucaderi
27bad14738 Added missing install target in newly added unit tsts 2020-09-22 08:42:50 +02:00
Luca Deri
60a9f6610d Added risks for checking
- invalid DNS traffic (probably carrying exfiltrated data)
- TLS traffic with no SNI extension
2020-09-21 19:57:23 +02:00
Alfredo Cardigliano
d700ab3994 Add distdir directive 2020-09-21 17:32:57 +02:00
Alfredo Cardigliano
e6d206fd15 Add unit tests to travis. Move ndpi serializer tests to unit tests. 2020-09-21 17:24:06 +02:00