Commit graph

46 commits

Author SHA1 Message Date
Toni Uhlig
9e07a57566
Major nDPId extension. Sorry for the huge commit.
- nDPId: fixed invalid IP4/IP6 tuple compare
 - nDPIsrvd: fixed caching issue (finally)
 - added tiny c example (can be used to check flow manager sanity)
 - c-captured: use flow_last_seen timestamp from `struct nDPIsrvd_flow`
 - README.md update: added example JSON sequence
 - nDPId: added new flow event `update` necessary for correct
   timeout handling (and other future use-cases)
 - nDPIsrvd.h and nDPIsrvd.py: switched to an instance
   (consists of an alias/source tuple) based flow manager
 - every flow related event **must** now serialize `alias`, `source`,
   `flow_id`, `flow_last_seen` and `flow_idle_time` to make the timeout
   handling and verification process work correctly
 - nDPIsrvd.h: ability to profile any dynamic memory (de-)allocation
 - nDPIsrvd.py: removed PcapPacket class (unused)
 - py-flow-dashboard and py-flow-multiprocess: fixed race condition
 - py-flow-info: print statusbar with probably useful information
 - nDPId/nDPIsrvd.h: switched from packet-flow only timestamps (`pkt_*sec`)
   to a generic flow event timestamp `ts_msec`
 - nDPId-test: added additional checks
 - nDPId: increased ICMP flow timeout
 - nDPId: using event based i/o if capturing packets from a device
 - nDPIsrvd: fixed memory leak on shutdown if remote descriptors
   were still connected

Signed-off-by: Toni Uhlig <matzeton@googlemail.com>
2022-01-20 00:50:38 +01:00
Toni Uhlig
d389f04135
MemoryProfiling: Advanced flow usage logging.
* nDPId-test: disable #include <syslog.h> if NO_MAIN macro defined
 * nDPId-test: mock syslog flags and functions
 * gitlab-ci: force -Werror

Signed-off-by: Toni Uhlig <matzeton@googlemail.com>
2021-11-13 17:30:21 +01:00
Toni Uhlig
9075706714
nDPId-test: Set max buffer size for remote descriptors useful to test caching/buffering.
Signed-off-by: Toni Uhlig <matzeton@googlemail.com>
2021-11-04 14:08:25 +01:00
Toni Uhlig
e54c2df63b
nDPIsrvd: Fixed anther bug, introduced during refactoring -_-
nDPId-test: Collect information about JSON string length's.

Signed-off-by: Toni Uhlig <matzeton@googlemail.com>
2021-09-15 14:33:13 +02:00
Toni Uhlig
aa89800ff9
fixed Warnings / build error / cosmetics
Signed-off-by: Toni Uhlig <matzeton@googlemail.com>
2021-09-14 18:38:37 +02:00
Toni Uhlig
6faded3cc7
Improved and Fixed another buffering issue caused by removing an outgoing fd too early from epoll queue (EPOLLOUT).
Signed-off-by: Toni Uhlig <matzeton@googlemail.com>
2021-08-05 02:02:51 +02:00
Toni Uhlig
d48508b4af
Improved nDPIsrvd buffer bloat handling using caching.
* still allow blocking mode (with send timeout)
 * improved daemon start/stop test script

Signed-off-by: Toni Uhlig <matzeton@googlemail.com>
2021-08-04 17:19:15 +02:00
Toni Uhlig
a41ddafa88
Git tag/commit version printing for nDPId/nDPIsrvd. Reduces confusion.
* disabled subshell spawn for run_tests.sh, common pitfall while using counters

Signed-off-by: Toni Uhlig <matzeton@googlemail.com>
2021-06-08 15:23:33 +02:00
Toni Uhlig
5954e46340
Build system cleanup / cosmetics.
* libnDPI submodule update

Signed-off-by: Toni Uhlig <matzeton@googlemail.com>
2021-06-07 16:22:49 +02:00
Toni Uhlig
54e0601fec
Unified IO buffer mgmt.
* c-collectd gives the user control over collectd-exec instance name
 * added missing collectd type `flow_l4_icmp_count`

Signed-off-by: Toni Uhlig <matzeton@googlemail.com>
2021-06-07 15:04:46 +02:00
Toni Uhlig
a0fa598cee
travis-ci build CMake project
Signed-off-by: Toni Uhlig <matzeton@googlemail.com>
2021-04-15 11:21:37 +02:00
Toni Uhlig
0a7ad7a76a
nDPId-test: added JSON distribution + JSON parsing (Multithreaded design re-using most of nDPId/nDPIsrvd core)
* improved Makefile.old install targets
 * splitted nDPIsrvd_parse into nDPIsrvd_parse_line and nDPIsrvd_parse_all for the sake of readability
 * minor Python script improvments (check for nDPIsrvd.py on multiple locations, may be superseeded by setuptools in the future)
 * some paths needs to be absolute (chdir() during daemonize) and therefor additional checks introduced
 * test run script checks and fails if certain files are are missing (PCAP file <=> result output file)
 * removed not very useful "internal format error" JSON serialization if a BUG for same exists
 * fixed invalid l4 type statistics counters for nDPIsrvd-collectd

Signed-off-by: Toni Uhlig <matzeton@googlemail.com>
2021-04-09 00:18:35 +02:00
Toni Uhlig
e835d36f63
Added nDPId-test as all-in-one JSON dumper.
* fixed invalid flow event schema type
 * added run_tests.sh to generate/diff JSON dumps
 * renamed lot's of vars/fns in nDPId.c/nDPIsrvd.c, so nDPId-test.c can include "*.c"
 * improved CMake dependency checks

Signed-off-by: Toni Uhlig <matzeton@googlemail.com>
2021-03-24 11:28:56 +01:00
Toni Uhlig
bdc8c5df2a
Reduced code duplication. Preps for nDPId-test.
Signed-off-by: Toni Uhlig <matzeton@googlemail.com>
2021-03-23 14:25:56 +01:00
Toni Uhlig
1c3ef69faa
nDPIsrvd collectd-exec overhaul.
* Install targets updated.
 * Removed nDPIsrvd.h token validation function (done automatically by token_get).

Signed-off-by: Toni Uhlig <matzeton@googlemail.com>
2021-03-15 14:39:43 +01:00
Toni Uhlig
1073c9626b
nDPIsrvd refactoring
Signed-off-by: Toni Uhlig <matzeton@googlemail.com>
2021-03-05 12:24:24 +01:00
Toni Uhlig
e0310d7e1d
Finalized examples/c-captured to dump packet bytes to PCAP for further analysis.
* Fixed memory holes in nDPId structs.

Signed-off-by: Toni Uhlig <matzeton@googlemail.com>
2021-02-24 18:43:26 +01:00
Toni Uhlig
0b5b177c14
Extended nDPIsrvd.h with address parsing.
* nDPId supports looading of custom nDPI protocol/category files
 * extended JSON schemas according to nDPI / nDPId JSON serializing
 * removed memory holes in nDPId
 * extended examples/c-captured

Signed-off-by: Toni Uhlig <matzeton@googlemail.com>
2021-02-23 14:46:47 +01:00
Toni Uhlig
257cce1dcc
Fixed braindead failure - Increase of NETWORK_BUFFER_MAX_SIZE > 9999 means to also increase NETWORK_BUFFER_LENGTH_DIGITS to 5.
* Fixed ARM32 xcompile warnings; Other GCC versions, other uint64_t's..
 * Replaced ridiculous nDPIsrvd_JSON_BYTES with NETWORK_BUFFER_LENGTH_DIGITS.

Signed-off-by: Toni Uhlig <matzeton@googlemail.com>
2021-01-27 18:05:13 +01:00
Toni Uhlig
9f3d7b479c
nDPIsrvd: Distributor clients which are too slow can cause buffer bloat.
Switching back to blocking mode works as a quick fix but is not sufficient.
          See comments.

 * nDPId prints more accurate error messages if command line argument validation failed

Signed-off-by: Toni Uhlig <matzeton@googlemail.com>
2020-12-19 20:51:21 +01:00
Toni Uhlig
f8692c6be1
Let travis use pcap files from nDPI to produce some JSON output. (disabled, needs further testing..)
* Added pcap diff script
 * Added \n to JSON string end (useful for debugging and readability)
 * Use first host/server name character for hash calculation as well
 * Removed error'ing EPOLLHUP handling in nDPIsrvd (connection closing will be detected via read())

Signed-off-by: Toni Uhlig <matzeton@googlemail.com>
2020-12-01 10:22:47 +01:00
Toni Uhlig
3cfb4d9c73
nDPId/nDPIsrvd: change_user_group does now chown/chmod collector/distributor socket paths
Signed-off-by: Toni Uhlig <matzeton@googlemail.com>
2020-10-07 14:38:10 +02:00
Toni Uhlig
1a7342e798
nDPIsrvd: Added very important warning and use UDS now by default.
Signed-off-by: Toni Uhlig <matzeton@googlemail.com>
2020-09-29 18:05:31 +02:00
Toni Uhlig
f60e7a097d
nDPIsrvd: Improved UDS related code parts.
Signed-off-by: Toni Uhlig <matzeton@googlemail.com>
2020-09-29 17:33:41 +02:00
Toni Uhlig
5e5c1a4d20
nDPIsrvd: Use of an anonymous enum for collector/distributor socket types fits best.
Signed-off-by: Toni Uhlig <matzeton@googlemail.com>
2020-09-07 21:05:47 +02:00
Toni Uhlig
ab374d193f
nDPIsrvd: Change user/group, allow listening on UNIX socket for incoming distributor connections
Signed-off-by: Toni Uhlig <matzeton@googlemail.com>
2020-09-05 17:52:34 +02:00
Toni Uhlig
1db474507b
nDPIsrvd: add command line option for distributor listen host/port
Signed-off-by: Toni Uhlig <matzeton@googlemail.com>
2020-08-18 17:14:05 +02:00
Toni Uhlig
230e96803e
nDPIsrvd: log ip:port for distributor connection fails
Signed-off-by: Toni Uhlig <matzeton@googlemail.com>
2020-08-17 16:10:43 +02:00
Toni Uhlig
9990865362
minor improvments regarding flow guessing on flow end/idle and other not worth to mention
Signed-off-by: Toni Uhlig <matzeton@googlemail.com>
2020-08-16 00:22:18 +02:00
Toni Uhlig
84448d5e4e
error handling enhancements
Signed-off-by: Toni Uhlig <matzeton@googlemail.com>
2020-08-15 13:05:49 +02:00
Toni Uhlig
a619a850c7
nDPIsrvd: fixed another two bugs; one related to EPOLLIN event for fd with shutdown reading end, one if write() did not write all bytes
Signed-off-by: Toni Uhlig <matzeton@googlemail.com>
2020-08-15 10:57:48 +02:00
Toni Uhlig
be4366b0e1
several fixes and improvments
- set errno to 0 if it is checked right after a libc call
  - ignore SIGPIPE as we want to avoid signal handling where possible
  - fixed another issue in nDPIsrvd/c-json-stdout which caused buffering errors

Signed-off-by: Toni Uhlig <matzeton@googlemail.com>
2020-08-14 17:40:02 +02:00
Toni Uhlig
a5f8783bda
minor improvments
Signed-off-by: Toni Uhlig <matzeton@googlemail.com>
2020-08-14 16:10:21 +02:00
Toni Uhlig
4b9b695946
nDPIsrvd: fixed broken buffering
Signed-off-by: Toni Uhlig <matzeton@googlemail.com>
2020-08-14 14:39:10 +02:00
Toni Uhlig
5e0a27d213
improved nDPIsrvd buffering if write returned EAGAIN
Signed-off-by: Toni Uhlig <matzeton@googlemail.com>
2020-08-12 12:24:39 +02:00
Toni Uhlig
8ccdadd3c7
do not disconnect a distributor client if write returned EAGAIN (still not perfect)
Signed-off-by: Toni Uhlig <matzeton@googlemail.com>
2020-08-09 23:07:06 +02:00
Toni Uhlig
5a7b0499ed
nDPIsrvd sighandler and cmd opt parser
Signed-off-by: Toni Uhlig <matzeton@googlemail.com>
2020-08-06 00:50:02 +02:00
Toni Uhlig
8a6021268e
introduced NETWORK_BUFFER_MAX_SIZE to replace BUFSIZ as this might change depending on the arch/libc used
Signed-off-by: Toni Uhlig <matzeton@googlemail.com>
2020-08-04 17:29:31 +02:00
Toni Uhlig
823b95828c
fixed buffering issue, improved logging verbosity
Signed-off-by: Toni Uhlig <matzeton@googlemail.com>
2020-08-04 16:22:00 +02:00
Toni Uhlig
3fd32fb337
clang-format
Signed-off-by: Toni Uhlig <matzeton@googlemail.com>
2020-08-04 10:28:56 +02:00
Toni Uhlig
bbeb147cde
nDPIsrvd: collect, buffer and distribute JSON strings
Signed-off-by: Toni Uhlig <matzeton@googlemail.com>
2020-08-04 10:27:18 +02:00
Toni Uhlig
88b85cecb8
merged some pre-processor macros which are used in multiple executables
Signed-off-by: Toni Uhlig <matzeton@googlemail.com>
2020-08-04 01:35:32 +02:00
Toni Uhlig
61807ffd09
forwarding data from collector(client,source,UNIX-sock) to distributor(client,sink,TCP-sock)
Signed-off-by: Toni Uhlig <matzeton@googlemail.com>
2020-08-03 17:12:30 +02:00
Toni Uhlig
92925a8355
remote connection tracking/ event-handling for collector(UNIX) and distributor(TCP) connections
Signed-off-by: Toni Uhlig <matzeton@googlemail.com>
2020-08-03 16:27:14 +02:00
Toni Uhlig
536a1c03a5
nDPIsrvd: getting events from epoll and accepting new connections
Signed-off-by: Toni Uhlig <matzeton@googlemail.com>
2020-07-19 21:02:35 +02:00
Toni Uhlig
89191dff42
nDPIsrvd: UNIX/INET socket creation
Signed-off-by: Toni Uhlig <matzeton@googlemail.com>
2020-07-19 16:53:00 +02:00