Commit graph

435 commits

Author SHA1 Message Date
Simone Mainardi
7320c8266c Adds throughput estimation via ZMQ
Implements #3508
2020-10-06 19:28:11 +02:00
Simone Mainardi
891e52c1ef Adds comments for flow status / alerted status 2020-10-02 17:03:47 +02:00
Simone Mainardi
fdf0f15140 Fixes races in view interfaces and cli/srv scores
Fixes #4513
Fixes #4512
2020-10-02 16:34:28 +02:00
Simone Mainardi
0db456c0cf Reworks flow user scripts execution in C++ 2020-09-30 18:41:36 +02:00
Simone Mainardi
90a5a57b36 Implements user script queues (avoids 5-sec HT walk) 2020-09-28 11:45:00 +02:00
Simone Mainardi
6e0c369fa9 Handles flow dump queues in view interfaces - and fixes races 2020-09-25 14:10:06 +02:00
Simone Mainardi
7f2a9b1420 Decouples flow dump from user scripts execution
Addresses #3545
2020-09-24 19:29:00 +02:00
Simone Mainardi
814ee67cf9 Reworks nDPI Risks flow plugins to handle all risks and scores
Fixes #4432
2020-09-23 17:58:51 +02:00
Luca Deri
55d8e299fa Cleanued up code for flow score 2020-09-23 10:51:36 +02:00
Luca Deri
c7b7f2ee1b Minor cleanup 2020-09-23 09:35:32 +02:00
Simone Mainardi
a9a323acf1 Reworks (simplifies) hosts score calculation
Fixes #4459
Fixes #4460
2020-09-22 15:46:38 +02:00
Simone Mainardi
841dacd4cd Simplifies per-category host score 2020-09-21 11:44:00 +02:00
Simone Mainardi
db0d7730a8 Implements per-category host score
Implements #4413
2020-09-18 18:34:28 +02:00
Luca Deri
6cd3ebcc3b Flow dump code rework 2020-09-11 15:58:33 +02:00
Luca Deri
4648dfffe1 Implemented full IEC 104 handling 2020-09-08 20:03:20 +02:00
Luca Deri
9fe21dac15 Added IEC 104 handling 2020-09-04 16:58:44 +02:00
Simone Mainardi
941da6819f Implements HTTP method using enum ndpi_http_method 2020-08-27 10:04:58 +02:00
Luca Deri
0bd6f1353a Added support for flow risk over ZMQ 2020-08-26 23:03:00 +02:00
Simone Mainardi
00c428eb1a Fixes heap-use-after-free on HTTP dissected last_url
Fixes #4328
2020-08-26 11:17:07 +02:00
Luca Deri
522fafc6df Added new flow call (to be used in scripts for instance) that allows from Lua to set a custom
info message.
Example: flow.setCustomInfo("hello world")
2020-08-04 19:00:40 +02:00
Simone Mainardi
21d151a404 Implements HTTP stats over ZMQ
Fixes #4152
2020-07-29 11:07:35 +02:00
Simone Mainardi
ec9d092423 Fixes DNS stats when collecting from ZMQ
Fixes #4148
2020-07-29 10:40:53 +02:00
Simone Mainardi
d33c1b23d7 Optimizes stats refresh calc for large flows 2020-07-27 18:51:20 +02:00
Simone Mainardi
1baa7d6cc4 Avoids double 5-sec HT-walks to update stats 2020-07-27 14:22:59 +02:00
Luca Deri
ce1309853f Refactored idle code: now it should be easier to read and understand.
Idle lifecycle has been modified to make it more efficient
2020-07-23 12:56:30 +02:00
Alfredo Cardigliano
b0c6ec0a84 Add tos to flow json (including elk) 2020-07-21 12:37:02 +02:00
Alfredo Cardigliano
0a84d0706e Add DSCP stats to the host, new DSCPStats class 2020-07-17 18:05:11 +02:00
Alfredo Cardigliano
2364e73345 DSCP utiliti functions 2020-07-17 10:40:38 +02:00
Luca Deri
1f2f1fd25d BA improvemements 2020-07-14 18:14:59 +02:00
Luca Deri
0fb01440b0 Implemented flow entropy 2020-07-14 12:49:47 +02:00
Alfredo Cardigliano
354866d16a Remove aggregated flows 2020-07-03 17:17:33 +02:00
Luca Deri
70c0601dfa Implemented DSCP/TOS collection, dissection and report 2020-06-25 23:23:04 +02:00
Simone Mainardi
d481465e72 Fixes flow exporter filtering
Fixes #4033
2020-06-12 10:59:23 +02:00
Simone Mainardi
4ffa3e2ca7 Adds SNMP information in flow_details.lua 2020-06-10 17:10:30 +02:00
Luca Deri
e87d13f260 Refreshed nDPI flow risk implementation 2020-05-16 00:26:09 +02:00
Simone Mainardi
058f2a32be Uses nDPI-calculated risks for TLS alerts 2020-05-15 11:54:18 +02:00
Luca Deri
d29492a89e Added support for flow risk reported by nDPI 2020-05-12 00:34:46 +02:00
Luca Deri
698265c4fb Added TLS issuerDN, subjectDN 2020-05-07 19:06:13 +02:00
Simone Mainardi
6082024c13 Updates flow throughput immediately for ZMQ flows
Possibly addresses #3863
2020-04-30 19:20:09 +02:00
Alfredo Cardigliano
823af75000 Propagate and print in flow details AS src/dst/prev/next from collected flows 2020-04-30 10:09:26 +00:00
Luca Deri
8ce11c8a48 Added flow.getnDPIMatchPacket() API call. See
https://github.com/ntop/ntopng/blob/dev/doc/src/api/lua_c/flow_user_scripts/flow.lua
for more info

Fixes #3797
2020-04-19 21:16:34 +02:00
Luca Deri
f99e184e90 Split LuaEngine.cpp apart (further improvements are necessary) 2020-04-19 12:29:13 +02:00
Simone Mainardi
43284e22f8 Fixes crash and removes unused PortContactStats
Class was not used and didn't handle concurrent accesses to strings,
yielding crashes such as

Thread 14 "stats_update.lu" received signal SIGSEGV, Segmentation fault.
[Switching to Thread 0x7fffb3fff700 (LWP 25268)]
0x00005555556b9183 in IpAddress::intoa (this=0x28c, buf=0x7fffb3ffe030 "", bufLen=64, bitmask=255 '\377') at src/IpAddress.cpp:352
352     src/IpAddress.cpp: No such file or directory.
(gdb)
(gdb) bt
    at src/GenericHash.cpp:222
    #9  0x00005555556c0413 in NetworkInterface::walker (this=0x555556159e70, begin_slot=0x7fffb3ffe454, walk_all=true, wtype=walker_flows, walker=0x5555556c6b79 <host_flow_update_stats(GenericHashEntry*, void*, bool*)>,
        user_data=0x7fffb3ffe458) at src/NetworkInterface.cpp:795
	#10 0x00005555556c6f7c in NetworkInterface::periodicStatsUpdate (this=0x555556159e70, vm=0x7fffac0a5578) at src/NetworkInterface.cpp:2589
	#11 0x0000555555671d62 in ntop_periodic_stats_update (vm=0x7fffac0a5578) at src/LuaEngine.cpp:6304
	#12 0x000055555576c2f6 in luaD_precall ()
	#13 0x0000555555777acd in luaV_execute ()
	#14 0x000055555576c5cf in luaD_call ()
	#15 0x000055555576c621 in luaD_callnoyield ()
	#16 0x000055555576ba42 in luaD_rawrunprotected ()
	#17 0x000055555576c91b in luaD_pcall ()
	#18 0x0000555555769cd4 in lua_pcallk ()
	#19 0x0000555555681e8a in LuaEngine::run_loaded_script (this=0x7fffac01fe10) at src/LuaEngine.cpp:12188
	#20 0x000055555563eca2 in ThreadedActivity::runScript (this=0x555562aee4e0, now=1585309410, script_path=0x7fff88002620 "/usr/share/ntopng/scripts/callbacks/interface/stats_update.lua", iface=0x555556159e70, deadline=1585309420)
	    at src/ThreadedActivity.cpp:418
	    #21 0x00005555556ba04d in ThreadPool::run (this=0x5555624525a0) at src/ThreadPool.cpp:103
	    #22 0x00005555556b9c79 in doRun (ptr=0x5555624525a0) at src/ThreadPool.cpp:31
	    #23 0x00007ffff65aa6db in start_thread (arg=0x7fffb3fff700) at pthread_create.c:463
	    #24 0x00007ffff3ea388f in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:95
2020-03-27 13:36:51 +01:00
Simone Mainardi
5bcd301a7d Allows nDPI dissection to continue on DNS flows
Fixes #3106
2020-03-12 19:09:48 +01:00
Simone Mainardi
88693f5f6a Adds flow tcp seq issues (ooo, retx, lost) over ZMQ 2020-03-04 15:01:24 +01:00
Alfredo Cardigliano
c3d3984285 Add more info to flow serialize (json) 2020-02-26 13:23:07 +01:00
emanuele-f
15898e8dad Flow dissection code cleanup.
The Flow API is now composed of the following methods:
  - Flow::setDetectedProtocol to manually set a protocol on the flow
    and terminate the dissection
  - Flow::processPacket to run the nDPI dissection on the raw packets
  - Flow::endProtocolDissection to terminate or give up the dissection

Also fixes the flow sampling rate preference, which was never used.
2020-02-17 17:36:06 +01:00
Luca Deri
8786c9fb30 Added support for ALPN and Client Supported TLS Protocols reported by nDPI 2020-02-17 16:29:24 +01:00
emanuele-f
107c9e50f2 Add shadow dns query to prevent use after free 2020-02-14 11:01:05 +01:00
Simone Mainardi
f7d79e50f9 Implements anti stall when dumping flows (MySQL, nIndex, ES, ...)
Implements #3411
2020-02-13 11:36:48 +01:00