Commit graph

477 commits

Author SHA1 Message Date
Toni Uhlig
c0b7bdacbc
Reworked nDPIsrvd.h C-API.
* nDPIsrvd.h: Provide nDPId thread storage.
 * nDPIsrvd.py: Fixed instance cleanup bug.
 * nDPIsrvd.h: Support for instance/thread user data and cleanup callback.
 * nDPIsrvd.h: Most recent flow time stored in thread ht instead of instance ht.
 * nDPId: Moved flow logger out the memory profilier into SIGUSR1 signal handling.
 * nDPId: Added signal fd to be usable within epoll's event handling (live-capture only!)
 * nDPId: Added information about ZLib compressions to daemon status/shutdown events.

Signed-off-by: Toni Uhlig <matzeton@googlemail.com>
2022-03-21 15:56:01 +01:00
Toni Uhlig
daaaa61519
Renamed basic event to error event for the sake of the logic.
Signed-off-by: Toni Uhlig <matzeton@googlemail.com>
2022-03-13 03:08:44 +01:00
Toni Uhlig
ed1647b944
Disconnect nDPIsrvd clients immediately instead waiting for a failed write().
* nDPIsrvd: Collector/Distributor logging improved
 * nDPIsrvd: Command line option for max remote descriptors
 * nDPId: Stop spamming nDPIsrvd Collector with the same events over and over again
 * nDPId: Refactored some variable names and events

Signed-off-by: Toni Uhlig <matzeton@googlemail.com>
2022-03-13 02:28:10 +01:00
Toni Uhlig
dd35d9da3f
CI: Fixed missing lcov prereq.
Signed-off-by: Toni Uhlig <matzeton@googlemail.com>
2022-03-12 11:17:03 +01:00
Toni Uhlig
f884a538ce
Code coverage generation using LCOV.
Signed-off-by: Toni Uhlig <matzeton@googlemail.com>
2022-03-11 18:29:38 +01:00
Toni Uhlig
41757ecf1c
Added nDPIsrvd TCP/IP support for distributors.
* nDPIsrvd: Improved distributor client disconnect detection
 * nDPIsrvd: Fixed invalid usage of epoll_add instead of epoll_mod
 * nPDIsrvd: Improved logging for distributor clients

Signed-off-by: Toni Uhlig <matzeton@googlemail.com>
2022-03-10 14:26:07 +01:00
Toni Uhlig
6f1f9e65ea
Fixed some pyhton issues with static class members.
Signed-off-by: Toni Uhlig <matzeton@googlemail.com>
2022-03-08 14:17:24 +01:00
Toni Uhlig
d0985a5732
Fixed build error regarding missing LINKTYPE_* define's.
Signed-off-by: Toni Uhlig <matzeton@googlemail.com>
2022-03-06 17:58:25 +01:00
Toni Uhlig
e09dd8509f
Updated examples/README.md
Signed-off-by: Toni Uhlig <matzeton@googlemail.com>
2022-03-06 17:41:38 +01:00
Toni Uhlig
29c72fb30b
Removed go-dashboard example.
Signed-off-by: Toni Uhlig <matzeton@googlemail.com>
2022-03-06 17:40:35 +01:00
Toni Uhlig
46f68501d5
Added daemon event: DAEMON_EVENT_STATUS (periodically send's daemon statistics.)
* Improved distributor timeout handling (per-thread).
 * flow-info.py / flow-dash.py: Distinguish between flow risk severities.
 * nDPId: Skip tag switch datalink packet dissection / processing.
 * nDPId: Fixed incorrect value for current active flows.
 * Improved JSON schema's.

Signed-off-by: Toni Uhlig <matzeton@googlemail.com>
2022-03-06 17:38:05 +01:00
Toni Uhlig
9db048c9d9
Serialize flow risk score / confidence.
* bump libnDPI to 8b062295cc76a60e3905c054ce37bd17669464d1
 * removed ndpi_id_struct's

Signed-off-by: Toni Uhlig <matzeton@googlemail.com>
2022-02-27 02:53:39 +01:00
Toni Uhlig
cb80c415d8
Improved py-flow-info to provide more optional information about received timestamps.
* py-flow-dashboard: Added color mapping for PieCharts/Graph that make more sense
 * nDPId: Renamed `flow_type' to a more precisely `flow_state'
 * nDPId: Changed the default setting to process only as much packets as libnDPI does

Signed-off-by: Toni Uhlig <matzeton@googlemail.com>
2022-02-05 15:27:13 +01:00
Toni Uhlig
6fd6dff14d
Added additional (minimalistic) detection information to flow updates.
This will only affect flows with the state `FT_FINISHED' (detection done).

 * nDPIsrvd.py: force use of JSON schema Draft 7 validator
 * flow-dash.py: gather/use total processed layer4 payload size
 * flow-info.py: added additional event filter
 * flow-info.py: prettified flow events printing whose detection is in progress
 * py-semantic-validation.py: added validation checks for FT_FINISHED
 * updated flow event JSON schema

Signed-off-by: Toni Uhlig <matzeton@googlemail.com>
2022-02-04 01:12:18 +01:00
Toni Uhlig
f9e4c58854
Added logging interface used by nDPId, nDPIsrvd and nDPId-test.
* fixed GitLab pipeline
 * nDPId: added static assert (just for a test)
 * nDPId: memory profiling for total bytes compressed
 * nDPId-test: enable zLib compression if configured with ENABLE_ZLIB

Signed-off-by: Toni Uhlig <matzeton@googlemail.com>
2022-02-03 03:48:37 +01:00
Toni Uhlig
1a0d7ddbfa
Process additional layer 3 protocols.
* bump libnDPI to c53c82d4823b5a8f856d1375155ac5112b68e8af
 * run_tests.sh: improved execution from non-git directories e.g. via `make dist`
 * updated JSON schema to be more restrictive
 * nDPId: splitted generic get_ip_from_sockaddr into IPv4/IPv6 to prevent compiler warnings on some platforms

Signed-off-by: Toni Uhlig <matzeton@googlemail.com>
2022-01-31 20:54:02 +01:00
Toni Uhlig
7022d0b1c5
nDPIsrvd: Fixed memory leak caused be not clearing buffer cache after a client disconnected.
* README.md: Fixed a typ0 and added a meh image from examples/py-flow-dashboard/flow-dash.py

Signed-off-by: Toni Uhlig <matzeton@googlemail.com>
2022-01-26 15:38:43 +01:00
Toni Uhlig
80e1eedbef
nDPId: Added some error messages when workflow init fails.
* Fixed invalid array subscript typ0 (caused some trouble..)
 * bump libnDPI to 2cd0479204301c50c6149706fcd4df3058b2a8cc

Signed-off-by: Toni Uhlig <matzeton@googlemail.com>
2022-01-26 15:12:28 +01:00
Toni Uhlig
4bae9d0344
py-flow-dashboard: added tab layout and event pie chart
Signed-off-by: Toni Uhlig <matzeton@googlemail.com>
2022-01-26 02:34:10 +01:00
Toni Uhlig
29a1b13e7a
Improved Plotly/Dash example. It is now somehow informative.
* TCP timeout after FIN/RST: switched back to the value from a35fc1d5ea
 * py-flow-info: reset 'guessed' flag after detection/detection-update received

Signed-off-by: Toni Uhlig <matzeton@googlemail.com>
2022-01-25 11:16:41 +01:00
Toni Uhlig
9e07a57566
Major nDPId extension. Sorry for the huge commit.
- nDPId: fixed invalid IP4/IP6 tuple compare
 - nDPIsrvd: fixed caching issue (finally)
 - added tiny c example (can be used to check flow manager sanity)
 - c-captured: use flow_last_seen timestamp from `struct nDPIsrvd_flow`
 - README.md update: added example JSON sequence
 - nDPId: added new flow event `update` necessary for correct
   timeout handling (and other future use-cases)
 - nDPIsrvd.h and nDPIsrvd.py: switched to an instance
   (consists of an alias/source tuple) based flow manager
 - every flow related event **must** now serialize `alias`, `source`,
   `flow_id`, `flow_last_seen` and `flow_idle_time` to make the timeout
   handling and verification process work correctly
 - nDPIsrvd.h: ability to profile any dynamic memory (de-)allocation
 - nDPIsrvd.py: removed PcapPacket class (unused)
 - py-flow-dashboard and py-flow-multiprocess: fixed race condition
 - py-flow-info: print statusbar with probably useful information
 - nDPId/nDPIsrvd.h: switched from packet-flow only timestamps (`pkt_*sec`)
   to a generic flow event timestamp `ts_msec`
 - nDPId-test: added additional checks
 - nDPId: increased ICMP flow timeout
 - nDPId: using event based i/o if capturing packets from a device
 - nDPIsrvd: fixed memory leak on shutdown if remote descriptors
   were still connected

Signed-off-by: Toni Uhlig <matzeton@googlemail.com>
2022-01-20 00:50:38 +01:00
Toni Uhlig
a35fc1d5ea
Removed py-flow-undetected-to-pcap and py-risky-flow-to-pcap. Done by c-captured anyway.
Signed-off-by: Toni Uhlig <matzeton@googlemail.com>
2021-11-21 12:01:45 +01:00
Toni Uhlig
cfecf3e110
go-dashboard renaming, ignore go-mod and it's file structure
Signed-off-by: Toni Uhlig <matzeton@googlemail.com>
2021-11-21 11:45:45 +01:00
Toni Uhlig
25b974af67
Use blocking I/O to prevent data loss if nDPIsrvd too slow.
* Fixed MemoryProfiler stack overflow.

Signed-off-by: Toni Uhlig <matzeton@googlemail.com>
2021-11-16 15:59:38 +01:00
Toni Uhlig
d389f04135
MemoryProfiling: Advanced flow usage logging.
* nDPId-test: disable #include <syslog.h> if NO_MAIN macro defined
 * nDPId-test: mock syslog flags and functions
 * gitlab-ci: force -Werror

Signed-off-by: Toni Uhlig <matzeton@googlemail.com>
2021-11-13 17:30:21 +01:00
Toni Uhlig
9075706714
nDPId-test: Set max buffer size for remote descriptors useful to test caching/buffering.
Signed-off-by: Toni Uhlig <matzeton@googlemail.com>
2021-11-04 14:08:25 +01:00
Toni Uhlig
1f6d1fbd67
Added timestamp validation test.
Signed-off-by: Toni Uhlig <matzeton@googlemail.com>
2021-11-02 12:15:41 +01:00
Toni Uhlig
d93c33aa74
Additional semantic validation tests.
Signed-off-by: Toni Uhlig <matzeton@googlemail.com>
2021-11-02 09:26:23 +01:00
Toni Uhlig
8ecd1b48ef
c-captured: Improved format string in nDPIsrvd_write_flow_info_cb.
Signed-off-by: Toni Uhlig <matzeton@googlemail.com>
2021-10-08 19:01:39 +02:00
Toni Uhlig
3af8de5a58
Fixed compile error due to missing stdint.h include before ndpi_typedefs.h
Signed-off-by: Toni Uhlig <matzeton@googlemail.com>
2021-10-08 11:42:27 +02:00
Toni Uhlig
315f90f982
Fixed invalid "flow_last_seen" timestamp for the first packet.
* After the first packet was processed, "flow_last_seen" was still 0.
   This behaviour is invalid as the first packet may contain l4 payload data e.g. for UDP
   and it also breaks nDPId json consistency "flow_first_seen" > 0, but "flow_last_seen" == 0.
 * JSON schema: set minimum timestamp value for Epoch timestamps to 24710 for flow_*_seen and
   1 for pcap packet ts. Those values are dependant on some manipulated pcap's in libnDPI/tests/pcap.

Signed-off-by: Toni Uhlig <matzeton@googlemail.com>
2021-10-08 11:31:58 +02:00
Toni Uhlig
fe77c44e3f
Added support/debug function to write flow(-user) related info.
Signed-off-by: Toni Uhlig <matzeton@googlemail.com>
2021-10-08 10:56:23 +02:00
Toni Uhlig
3726311276
bump libnDPI to 181a03c5ad41bda533fbfa307627939c2ff30b75
Signed-off-by: Toni Uhlig <matzeton@googlemail.com>
2021-10-05 23:39:11 +02:00
Toni Uhlig
a523c348f3
More CMake warnings/errors/fixes added.
Signed-off-by: Toni Uhlig <matzeton@googlemail.com>
2021-09-29 18:40:12 +02:00
Toni Uhlig
5a6b2aa261
CMake and CI extensions
* CPack support for debian packages
 * Use CPack version string for nDPId

Signed-off-by: Toni Uhlig <matzeton@googlemail.com>
2021-09-29 15:46:47 +02:00
Toni Uhlig
992d3a207d
dumb fuzzer: randpkt vs nDPId-test
Signed-off-by: Toni Uhlig <matzeton@googlemail.com>
2021-09-20 00:28:44 +02:00
Toni Uhlig
7829bfe4e6
CI extended and fixups
Signed-off-by: Toni Uhlig <matzeton@googlemail.com>
2021-09-19 11:30:55 +02:00
Toni Uhlig
4fa1694b05
Github Actions integration
Signed-off-by: Toni Uhlig <matzeton@googlemail.com>
2021-09-17 18:59:49 +02:00
Toni Uhlig
c5be804725
Removed Travis-CI support as they do not support OpenSource anymore.
Signed-off-by: Toni Uhlig <matzeton@googlemail.com>
2021-09-16 16:48:55 +02:00
Toni Uhlig
655f38b68f
Fixed some typ0's and reduced ICMP timeout to 10s.
* nDPId: Renamed some of the misleading terms, still TODO for nDPIsrvd
 * CMake improvments

Signed-off-by: Toni Uhlig <matzeton@googlemail.com>
2021-09-16 16:43:30 +02:00
Toni Uhlig
4edf3bf7e6
Merge commit '1fa53c5bf8' 2021-09-15 17:04:21 +02:00
Toni Uhlig
1fa53c5bf8 Squashed 'dependencies/uthash/' changes from 8e67ced..bf15263
bf15263 Fix a "bug" in the example where option 3 interfered with option 1's counter.
b6e24ef Use `malloc(sizeof *s)` in example code.
a109c6b Stop using `gets` in example.c.
c85c9e1 fix: fix utstack example's compiling error
86e6776 Replace *.github.com urls with *.github.io (#227)
e493aa9 Bump version to 2.3.0.
ae2ac52 Fix README.md to display the *actual* TravisCI status.
134e241 Silence -Wswitch-default warnings, and add it to the TravisCI config.
62fefa6 Fix some typos in userguide.txt, and re-remove spaces in macro definitions.
37d2021 tests: add whitespaces to example code
524ca1a doc: add whitespaces to documentation
0f6c619 Fix a typo in the documentation for HASH_COUNT. NFC.
388134a Rename uthash_memcmp to HASH_KEYCMP, step 3.
053bed1 Eliminate HASH_FCN; change the handling of HASH_FUNCTION to match HASH_KEYCMP.
f0e1bd9 Refactor test93.c to avoid scan-build warnings.
45af88c Remove two dead writes in tests, to silence scan-build warnings.
66e2668 Bump version to 2.2.0.
973bd67 uthash.h: Swap multiplicands to put the widest ones first.
15ad042 Always include <stdint.h>, unless HASH_NO_STDINT is defined by the user.
6b4768b Rename uthash_memcmp to HASH_KEYCMP, step 2.
e64c7f0 Update tests/README to describe the most recently added tests. NFC.
c62796c HASH_CLEAR after some tests, to eliminate "memory leak" warnings.
7f0aadb Support spaces in $exe path
0831d9a uthash.h: fix compiler warning -Wcast-qual
ba2fbfd utarray.h: preserve constness in utarray_str_cpy

git-subtree-dir: dependencies/uthash
git-subtree-split: bf15263081be6229be31addd48566df93921cb46
2021-09-15 17:04:21 +02:00
Toni Uhlig
2a5e5a020b
Merge commit '8e096b19c1' 2021-09-15 17:03:59 +02:00
Toni Uhlig
8e096b19c1 Squashed 'dependencies/jsmn/' changes from 053d3cd..1aa2e8f
1aa2e8f Update README.md (#203)
b85f161 Update README.md (#213)
23f13d2 Merge pull request #108 from olmokramer/patch-1

git-subtree-dir: dependencies/jsmn
git-subtree-split: 1aa2e8f80849c983466b165d53542da9b1bd1b32
2021-09-15 17:03:59 +02:00
Toni Uhlig
e54c2df63b
nDPIsrvd: Fixed anther bug, introduced during refactoring -_-
nDPId-test: Collect information about JSON string length's.

Signed-off-by: Toni Uhlig <matzeton@googlemail.com>
2021-09-15 14:33:13 +02:00
Toni Uhlig
c152e41cfb
README.md ascii update
Signed-off-by: Toni Uhlig <matzeton@googlemail.com>
2021-09-14 18:54:33 +02:00
Toni Uhlig
aa89800ff9
fixed Warnings / build error / cosmetics
Signed-off-by: Toni Uhlig <matzeton@googlemail.com>
2021-09-14 18:38:37 +02:00
Toni Uhlig
ea0b04d648
bump libnDPI to 0eb7a0388c4549ebbf8cd7a10d398088005cc2de
Signed-off-by: Toni Uhlig <matzeton@googlemail.com>
2021-09-14 18:19:47 +02:00
Toni Uhlig
6faded3cc7
Improved and Fixed another buffering issue caused by removing an outgoing fd too early from epoll queue (EPOLLOUT).
Signed-off-by: Toni Uhlig <matzeton@googlemail.com>
2021-08-05 02:02:51 +02:00
Toni Uhlig
d48508b4af
Improved nDPIsrvd buffer bloat handling using caching.
* still allow blocking mode (with send timeout)
 * improved daemon start/stop test script

Signed-off-by: Toni Uhlig <matzeton@googlemail.com>
2021-08-04 17:19:15 +02:00