Commit graph

27 commits

Author SHA1 Message Date
Toni Uhlig
29a1b13e7a
Improved Plotly/Dash example. It is now somehow informative.
* TCP timeout after FIN/RST: switched back to the value from a35fc1d5ea
 * py-flow-info: reset 'guessed' flag after detection/detection-update received

Signed-off-by: Toni Uhlig <matzeton@googlemail.com>
2022-01-25 11:16:41 +01:00
Toni Uhlig
9e07a57566
Major nDPId extension. Sorry for the huge commit.
- nDPId: fixed invalid IP4/IP6 tuple compare
 - nDPIsrvd: fixed caching issue (finally)
 - added tiny c example (can be used to check flow manager sanity)
 - c-captured: use flow_last_seen timestamp from `struct nDPIsrvd_flow`
 - README.md update: added example JSON sequence
 - nDPId: added new flow event `update` necessary for correct
   timeout handling (and other future use-cases)
 - nDPIsrvd.h and nDPIsrvd.py: switched to an instance
   (consists of an alias/source tuple) based flow manager
 - every flow related event **must** now serialize `alias`, `source`,
   `flow_id`, `flow_last_seen` and `flow_idle_time` to make the timeout
   handling and verification process work correctly
 - nDPIsrvd.h: ability to profile any dynamic memory (de-)allocation
 - nDPIsrvd.py: removed PcapPacket class (unused)
 - py-flow-dashboard and py-flow-multiprocess: fixed race condition
 - py-flow-info: print statusbar with probably useful information
 - nDPId/nDPIsrvd.h: switched from packet-flow only timestamps (`pkt_*sec`)
   to a generic flow event timestamp `ts_msec`
 - nDPId-test: added additional checks
 - nDPId: increased ICMP flow timeout
 - nDPId: using event based i/o if capturing packets from a device
 - nDPIsrvd: fixed memory leak on shutdown if remote descriptors
   were still connected

Signed-off-by: Toni Uhlig <matzeton@googlemail.com>
2022-01-20 00:50:38 +01:00
Toni Uhlig
655f38b68f
Fixed some typ0's and reduced ICMP timeout to 10s.
* nDPId: Renamed some of the misleading terms, still TODO for nDPIsrvd
 * CMake improvments

Signed-off-by: Toni Uhlig <matzeton@googlemail.com>
2021-09-16 16:43:30 +02:00
Toni Uhlig
d48508b4af
Improved nDPIsrvd buffer bloat handling using caching.
* still allow blocking mode (with send timeout)
 * improved daemon start/stop test script

Signed-off-by: Toni Uhlig <matzeton@googlemail.com>
2021-08-04 17:19:15 +02:00
Toni Uhlig
db87d45edb
Added zLib compression parameters to control compression conditions.
* more structs are now "compressable"
 * fixed missing DAEMON_RECONNECT event
 * improved memory profiler

Signed-off-by: Toni Uhlig <matzeton@googlemail.com>
2021-06-16 19:28:02 +02:00
Toni Uhlig
da4942b41c
Use layer4 specific flow timeouts.
* default values "stolen" from nf_conntrack

Signed-off-by: Toni Uhlig <matzeton@googlemail.com>
2021-05-13 15:41:24 +02:00
Toni Uhlig
e576162a43
Reverted internal ndpi structs to dynamic memory allocation hoping that it will help to reduce the average memory consumption.
How? After the detection finished, internal ndpi structs can be free'd as they are not needed anymore.

 * Set the amount of max. packets to process via subopt.

Signed-off-by: Toni Uhlig <matzeton@googlemail.com>
2021-03-26 14:46:35 +01:00
Toni Uhlig
69b4e662c7
nDPIsrvd C API overhaul and massive simplification.
* nDPIsrvd.h does flow mgmt out of the box
 * dissect received JSON strings via callback
 * added new JSON key/values for packet-flows (usecTimestamp/L3/L4 info)

Signed-off-by: Toni Uhlig <matzeton@googlemail.com>
2021-02-06 15:41:30 +01:00
Toni Uhlig
257cce1dcc
Fixed braindead failure - Increase of NETWORK_BUFFER_MAX_SIZE > 9999 means to also increase NETWORK_BUFFER_LENGTH_DIGITS to 5.
* Fixed ARM32 xcompile warnings; Other GCC versions, other uint64_t's..
 * Replaced ridiculous nDPIsrvd_JSON_BYTES with NETWORK_BUFFER_LENGTH_DIGITS.

Signed-off-by: Toni Uhlig <matzeton@googlemail.com>
2021-01-27 18:05:13 +01:00
Toni Uhlig
9564b0ce2c
Increased JSON buffer size to 12288 (libnDPI serializes more and more information).
* Making Compare&Fetch mandatory.
 * Added some more Compare&Fetch to prevent TSAN complaining about data races.
   Fixed possible but more ore less harmless data races during shutdown process.
 * Shrink SIGNAL handler to a minimum. SYSV Signal handling and MT-safety is awkward.

Signed-off-by: Toni Uhlig <matzeton@googlemail.com>
2021-01-27 17:19:29 +01:00
Toni Uhlig
f5d5c076a3
Added internal/external packet processing modes. #2
* fixed incorrect handling of skipped flows

Signed-off-by: Toni Uhlig <matzeton@googlemail.com>
2021-01-10 19:12:05 +01:00
Toni Uhlig
b82a535a45
Increased network buffer size for python scripts (and raise exception if capacity reached).
Signed-off-by: Toni Uhlig <matzeton@googlemail.com>
2021-01-08 10:54:44 +01:00
Toni Uhlig
b37cafe910
Improved testing capabilities via Makefile.
* increased nDPId_MAX_READER_THREADS to 32

Signed-off-by: Toni Uhlig <matzeton@googlemail.com>
2020-12-15 10:32:16 +01:00
Toni Uhlig
ba179cc724
Increased serialization buffer size as required for recent nDPI builds.
* added .gitignore to prevent SCM check-ins of unwanted files
 * fixed invalid include path for python examples

Signed-off-by: Toni Uhlig <matzeton@googlemail.com>
2020-11-18 18:12:38 +01:00
Toni Uhlig
1a7342e798
nDPIsrvd: Added very important warning and use UDS now by default.
Signed-off-by: Toni Uhlig <matzeton@googlemail.com>
2020-09-29 18:05:31 +02:00
Toni Uhlig
6c0ac8b045
Added new flow event: FLOW_EVENT_DETECTION_UPDATE
* This event will be triggered when nDPI detection has some new information for us (hopefully).
 * Detection change is based on hashing with 32-bit murmur3 certain members of the ndpi flow struct.

Signed-off-by: Toni Uhlig <matzeton@googlemail.com>
2020-09-25 20:43:23 +02:00
Toni Uhlig
ab374d193f
nDPIsrvd: Change user/group, allow listening on UNIX socket for incoming distributor connections
Signed-off-by: Toni Uhlig <matzeton@googlemail.com>
2020-09-05 17:52:34 +02:00
Toni Uhlig
905d84506e
Increased network buffer size from 8448 to 9216 with the hope that it might be finally enough.
Signed-off-by: Toni Uhlig <matzeton@googlemail.com>
2020-08-27 15:40:43 +02:00
Toni Uhlig
93fa7fcabf
nDPId: Improved command line option parsing, app usage and subopts for (carefully) tuning some daemon options.
Signed-off-by: Toni Uhlig <matzeton@googlemail.com>
2020-08-26 17:24:12 +02:00
Toni Uhlig
e8a115b39f
adjust some config values
Signed-off-by: Toni Uhlig <matzeton@googlemail.com>
2020-08-15 10:48:38 +02:00
Toni Uhlig
a377008644
nDPId: removed unused code, process remaining flows on shutdown (useful for replaying pcap files)
Signed-off-by: Toni Uhlig <matzeton@googlemail.com>
2020-08-14 14:37:50 +02:00
Toni Uhlig
3f783f9f01
improved TCP-FIN/TCP-RST and TCP-keepalive/-idle timeout handling
Signed-off-by: Toni Uhlig <matzeton@googlemail.com>
2020-08-13 09:49:14 +02:00
Toni Uhlig
5a7b0499ed
nDPIsrvd sighandler and cmd opt parser
Signed-off-by: Toni Uhlig <matzeton@googlemail.com>
2020-08-06 00:50:02 +02:00
Toni Uhlig
88aa768184
nDPId daemonize / pidfile support + improved syslog logging
Signed-off-by: Toni Uhlig <matzeton@googlemail.com>
2020-08-05 12:05:32 +02:00
Toni Uhlig
8a6021268e
introduced NETWORK_BUFFER_MAX_SIZE to replace BUFSIZ as this might change depending on the arch/libc used
Signed-off-by: Toni Uhlig <matzeton@googlemail.com>
2020-08-04 17:29:31 +02:00
Toni Uhlig
913c8d5a18
moved more default config options into config.h
Signed-off-by: Toni Uhlig <matzeton@googlemail.com>
2020-08-04 01:55:21 +02:00
Toni Uhlig
88b85cecb8
merged some pre-processor macros which are used in multiple executables
Signed-off-by: Toni Uhlig <matzeton@googlemail.com>
2020-08-04 01:35:32 +02:00