Commit graph

27 commits

Author SHA1 Message Date
Toni Uhlig
14f6b87551
Added nDPIsrvd-analysed to generate CSV files from analyse events.
* nDPIsrvd.h: iterate over JSON arrays
 * nDPId: calculate l3 payload packet entropies for analysis

Signed-off-by: Toni Uhlig <matzeton@googlemail.com>
2022-09-30 19:28:49 +02:00
Toni Uhlig
2103ee0811
Refactored client distributor C API.
* Still not perfect, but the code before was not even able to deal with JSON arrays.
   Use common "speaking" function names for all functions in nDPIsrvd.h
 * Provide a more or less generic and easy extendable JSON walk function.
 * Modified C examples to align with the changed C API.
 * c-collectd: Reduced lot's of code duplication by providing mapping tables.
 * nDPId: IAT array requires one slot less (first packet has always an IAT of 0).

Signed-off-by: Toni Uhlig <matzeton@googlemail.com>
2022-09-25 00:54:39 +02:00
Toni Uhlig
9a28475bba
Improved flown analyse event:
* store packet directions
 * merged direction based IATs
 * merged direction based PKTLENs

Signed-off-by: Toni Uhlig <matzeton@googlemail.com>
2022-09-22 19:07:08 +02:00
Toni Uhlig
08f263e409
nDPId: Reduced flow-updates for TCP flows to 1/4 of the timeout value.
* nDPId: Fixed broken validation tests.
 * nDPId: Removed TICK_RESOLUTION, not required anymore.
 * c-collectd: Improved total layer4 payload calculation/update handling.
 * c-collectd: Updated RRD Graph script according to total layer4 payload changes.
 * py-flow-info.py: Fixed several bugs and syntax errors.
 * Python scripts: Added dirname(argv[0]) as search path for nDPIsrvd.py.
 * nDPIsrvd&nDPId-test: Fixed missing EPOLLERR check.

Signed-off-by: Toni Uhlig <matzeton@googlemail.com>
2022-09-19 19:39:49 +02:00
Toni Uhlig
015a739efd
Added layer4 payload length bins.
Signed-off-by: Toni Uhlig <matzeton@googlemail.com>
2022-09-19 10:14:37 +02:00
Toni Uhlig
31715295d9
bump libnDPI to 174cd739dbb1358ab012c4779e42e0221bef835c
* ReadMe stuff
 * OpenWrt Makefile: set NEED_LINKING_AGAINST_LIBM=ON

Signed-off-by: Toni Uhlig <matzeton@googlemail.com>
2022-09-18 13:34:43 +02:00
Toni Uhlig
d4633c1192
New flow event: 'analysis'.
* The goal was to provide a separate event for extracted feature that are not required
   and only useful for a few (e.g. someone who wants do ML).
 * Increased network buffer size to 32kB (8192 * 4).
 * Switched timestamp precision from ms to us for *ALL* timestamps.

Signed-off-by: Toni Uhlig <matzeton@googlemail.com>
2022-09-13 22:05:08 +02:00
Toni Uhlig
f9bd7d29ce
Bump libnDPI to 37f918322c0a489b5143a987c8f1a44a6f78a6f3 and updated flow json schema file.
* export env vars AR / CMAKE_C_COMPILER_AR and RANLIB / CMAKE_C_COMPILER_RANLIB while building libnDPI
 * nDPId check API version during startup (macro vs. function call) and print a warning if they are different

Signed-off-by: Toni Uhlig <matzeton@googlemail.com>
2022-09-06 14:50:46 +02:00
Toni Uhlig
0fd59f060e
Split *_l4_payload_len' into *_src_l4_payload_len' and `*_dst_l4_payload_len'.
Signed-off-by: Toni Uhlig <matzeton@googlemail.com>
2022-08-15 22:55:19 +02:00
lns
905545487d Split flow_packets_processed' into flow_src_packets_processed' and `flow_dst_packets_processed'.
* no use for `flow_avg_l4_payload_len' -> removed
 * test/run_tests.sh does not fail if git-worktree's are used

Signed-off-by: lns <matzeton@googlemail.com>
2022-08-15 18:36:49 +02:00
Toni Uhlig
a606586a32
bump libnDPI to 7c19de49047a5731f3107ff17854e9afe839cc61
Signed-off-by: Toni Uhlig <matzeton@googlemail.com>
2022-07-14 03:48:06 +02:00
Toni Uhlig
b5d4da8793
bump libnDPI to 8f6a006e36eef0ae386f7e663d3ebecfad6a2dc9
* try to use same wording wherever possible e.g.
   renamed workflow->total_l4_data_len to workflow->total_l4_payload_len

Signed-off-by: Toni Uhlig <matzeton@googlemail.com>
2022-07-01 13:50:53 +02:00
Toni Uhlig
a80b6d7271
bump libnDPI to c287eb835b537ce64d9293a52ca13e670b6d3b0d
Signed-off-by: Toni Uhlig <matzeton@googlemail.com>
2022-06-19 23:29:26 +02:00
lns
3caf7727fd bump libnDPI to 0b3f8ed849cdf9971224c49a3958f0904a2bbbb5
* README/nDPId: fixed typ0

Signed-off-by: lns <matzeton@googlemail.com>
2022-06-06 00:34:13 +02:00
lns
4fde63b5c2 Small fixes.
Signed-off-by: lns <matzeton@googlemail.com>
2022-04-26 11:44:31 +02:00
lns
a46fc4153d nDPId: Merged nDPId_flow_(info|finished) into nDPId_flow
* nDPIsrvd: Fixed buffer allocation error due to missing memset() on disconnect
 * nDPIsrvd: Removed unused struct members

Signed-off-by: lns <matzeton@googlemail.com>
2022-04-24 23:49:57 +02:00
lns
9aeff586bd bump libnDPI to 8b2c9860be8b0663bfe9fc3b6defc041bb90e5b2
Signed-off-by: lns <matzeton@googlemail.com>
2022-04-18 19:26:27 +02:00
Toni Uhlig
9f9e881b3f
bump libnDPI to bb12837ca75efc2691ecb18fd5f56e2d097ef26b
Signed-off-by: Toni Uhlig <matzeton@googlemail.com>
2022-03-24 02:16:33 +01:00
Toni Uhlig
65a9e5a18d
Executing ./tests/run_tests.sh w/o zLib should not result in diff's anymore.
Signed-off-by: Toni Uhlig <matzeton@googlemail.com>
2022-03-24 01:04:49 +01:00
Toni Uhlig
c0b7bdacbc
Reworked nDPIsrvd.h C-API.
* nDPIsrvd.h: Provide nDPId thread storage.
 * nDPIsrvd.py: Fixed instance cleanup bug.
 * nDPIsrvd.h: Support for instance/thread user data and cleanup callback.
 * nDPIsrvd.h: Most recent flow time stored in thread ht instead of instance ht.
 * nDPId: Moved flow logger out the memory profilier into SIGUSR1 signal handling.
 * nDPId: Added signal fd to be usable within epoll's event handling (live-capture only!)
 * nDPId: Added information about ZLib compressions to daemon status/shutdown events.

Signed-off-by: Toni Uhlig <matzeton@googlemail.com>
2022-03-21 15:56:01 +01:00
Toni Uhlig
46f68501d5
Added daemon event: DAEMON_EVENT_STATUS (periodically send's daemon statistics.)
* Improved distributor timeout handling (per-thread).
 * flow-info.py / flow-dash.py: Distinguish between flow risk severities.
 * nDPId: Skip tag switch datalink packet dissection / processing.
 * nDPId: Fixed incorrect value for current active flows.
 * Improved JSON schema's.

Signed-off-by: Toni Uhlig <matzeton@googlemail.com>
2022-03-06 17:38:05 +01:00
Toni Uhlig
9db048c9d9
Serialize flow risk score / confidence.
* bump libnDPI to 8b062295cc76a60e3905c054ce37bd17669464d1
 * removed ndpi_id_struct's

Signed-off-by: Toni Uhlig <matzeton@googlemail.com>
2022-02-27 02:53:39 +01:00
Toni Uhlig
cb80c415d8
Improved py-flow-info to provide more optional information about received timestamps.
* py-flow-dashboard: Added color mapping for PieCharts/Graph that make more sense
 * nDPId: Renamed `flow_type' to a more precisely `flow_state'
 * nDPId: Changed the default setting to process only as much packets as libnDPI does

Signed-off-by: Toni Uhlig <matzeton@googlemail.com>
2022-02-05 15:27:13 +01:00
Toni Uhlig
6fd6dff14d
Added additional (minimalistic) detection information to flow updates.
This will only affect flows with the state `FT_FINISHED' (detection done).

 * nDPIsrvd.py: force use of JSON schema Draft 7 validator
 * flow-dash.py: gather/use total processed layer4 payload size
 * flow-info.py: added additional event filter
 * flow-info.py: prettified flow events printing whose detection is in progress
 * py-semantic-validation.py: added validation checks for FT_FINISHED
 * updated flow event JSON schema

Signed-off-by: Toni Uhlig <matzeton@googlemail.com>
2022-02-04 01:12:18 +01:00
Toni Uhlig
f9e4c58854
Added logging interface used by nDPId, nDPIsrvd and nDPId-test.
* fixed GitLab pipeline
 * nDPId: added static assert (just for a test)
 * nDPId: memory profiling for total bytes compressed
 * nDPId-test: enable zLib compression if configured with ENABLE_ZLIB

Signed-off-by: Toni Uhlig <matzeton@googlemail.com>
2022-02-03 03:48:37 +01:00
Toni Uhlig
1a0d7ddbfa
Process additional layer 3 protocols.
* bump libnDPI to c53c82d4823b5a8f856d1375155ac5112b68e8af
 * run_tests.sh: improved execution from non-git directories e.g. via `make dist`
 * updated JSON schema to be more restrictive
 * nDPId: splitted generic get_ip_from_sockaddr into IPv4/IPv6 to prevent compiler warnings on some platforms

Signed-off-by: Toni Uhlig <matzeton@googlemail.com>
2022-01-31 20:54:02 +01:00
Toni Uhlig
80e1eedbef
nDPId: Added some error messages when workflow init fails.
* Fixed invalid array subscript typ0 (caused some trouble..)
 * bump libnDPI to 2cd0479204301c50c6149706fcd4df3058b2a8cc

Signed-off-by: Toni Uhlig <matzeton@googlemail.com>
2022-01-26 15:12:28 +01:00