Commit graph

762 commits

Author SHA1 Message Date
Simone Mainardi
1baa7d6cc4 Avoids double 5-sec HT-walks to update stats 2020-07-27 14:22:59 +02:00
Luca Deri
ce1309853f Refactored idle code: now it should be easier to read and understand.
Idle lifecycle has been modified to make it more efficient
2020-07-23 12:56:30 +02:00
Alfredo Cardigliano
b0c6ec0a84 Add tos to flow json (including elk) 2020-07-21 12:37:02 +02:00
Alfredo Cardigliano
0a84d0706e Add DSCP stats to the host, new DSCPStats class 2020-07-17 18:05:11 +02:00
Alfredo Cardigliano
2364e73345 DSCP utiliti functions 2020-07-17 10:40:38 +02:00
Luca Deri
1f2f1fd25d BA improvemements 2020-07-14 18:14:59 +02:00
Luca Deri
0fb01440b0 Implemented flow entropy 2020-07-14 12:49:47 +02:00
Alfredo Cardigliano
354866d16a Remove aggregated flows 2020-07-03 17:17:33 +02:00
Luca Deri
70c0601dfa Implemented DSCP/TOS collection, dissection and report 2020-06-25 23:23:04 +02:00
Simone Mainardi
d481465e72 Fixes flow exporter filtering
Fixes #4033
2020-06-12 10:59:23 +02:00
Simone Mainardi
4ffa3e2ca7 Adds SNMP information in flow_details.lua 2020-06-10 17:10:30 +02:00
Luca Deri
e87d13f260 Refreshed nDPI flow risk implementation 2020-05-16 00:26:09 +02:00
Simone Mainardi
058f2a32be Uses nDPI-calculated risks for TLS alerts 2020-05-15 11:54:18 +02:00
Luca Deri
d29492a89e Added support for flow risk reported by nDPI 2020-05-12 00:34:46 +02:00
Luca Deri
698265c4fb Added TLS issuerDN, subjectDN 2020-05-07 19:06:13 +02:00
Simone Mainardi
6082024c13 Updates flow throughput immediately for ZMQ flows
Possibly addresses #3863
2020-04-30 19:20:09 +02:00
Alfredo Cardigliano
823af75000 Propagate and print in flow details AS src/dst/prev/next from collected flows 2020-04-30 10:09:26 +00:00
Luca Deri
8ce11c8a48 Added flow.getnDPIMatchPacket() API call. See
https://github.com/ntop/ntopng/blob/dev/doc/src/api/lua_c/flow_user_scripts/flow.lua
for more info

Fixes #3797
2020-04-19 21:16:34 +02:00
Luca Deri
f99e184e90 Split LuaEngine.cpp apart (further improvements are necessary) 2020-04-19 12:29:13 +02:00
Simone Mainardi
43284e22f8 Fixes crash and removes unused PortContactStats
Class was not used and didn't handle concurrent accesses to strings,
yielding crashes such as

Thread 14 "stats_update.lu" received signal SIGSEGV, Segmentation fault.
[Switching to Thread 0x7fffb3fff700 (LWP 25268)]
0x00005555556b9183 in IpAddress::intoa (this=0x28c, buf=0x7fffb3ffe030 "", bufLen=64, bitmask=255 '\377') at src/IpAddress.cpp:352
352     src/IpAddress.cpp: No such file or directory.
(gdb)
(gdb) bt
    at src/GenericHash.cpp:222
    #9  0x00005555556c0413 in NetworkInterface::walker (this=0x555556159e70, begin_slot=0x7fffb3ffe454, walk_all=true, wtype=walker_flows, walker=0x5555556c6b79 <host_flow_update_stats(GenericHashEntry*, void*, bool*)>,
        user_data=0x7fffb3ffe458) at src/NetworkInterface.cpp:795
	#10 0x00005555556c6f7c in NetworkInterface::periodicStatsUpdate (this=0x555556159e70, vm=0x7fffac0a5578) at src/NetworkInterface.cpp:2589
	#11 0x0000555555671d62 in ntop_periodic_stats_update (vm=0x7fffac0a5578) at src/LuaEngine.cpp:6304
	#12 0x000055555576c2f6 in luaD_precall ()
	#13 0x0000555555777acd in luaV_execute ()
	#14 0x000055555576c5cf in luaD_call ()
	#15 0x000055555576c621 in luaD_callnoyield ()
	#16 0x000055555576ba42 in luaD_rawrunprotected ()
	#17 0x000055555576c91b in luaD_pcall ()
	#18 0x0000555555769cd4 in lua_pcallk ()
	#19 0x0000555555681e8a in LuaEngine::run_loaded_script (this=0x7fffac01fe10) at src/LuaEngine.cpp:12188
	#20 0x000055555563eca2 in ThreadedActivity::runScript (this=0x555562aee4e0, now=1585309410, script_path=0x7fff88002620 "/usr/share/ntopng/scripts/callbacks/interface/stats_update.lua", iface=0x555556159e70, deadline=1585309420)
	    at src/ThreadedActivity.cpp:418
	    #21 0x00005555556ba04d in ThreadPool::run (this=0x5555624525a0) at src/ThreadPool.cpp:103
	    #22 0x00005555556b9c79 in doRun (ptr=0x5555624525a0) at src/ThreadPool.cpp:31
	    #23 0x00007ffff65aa6db in start_thread (arg=0x7fffb3fff700) at pthread_create.c:463
	    #24 0x00007ffff3ea388f in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:95
2020-03-27 13:36:51 +01:00
Simone Mainardi
5bcd301a7d Allows nDPI dissection to continue on DNS flows
Fixes #3106
2020-03-12 19:09:48 +01:00
Simone Mainardi
88693f5f6a Adds flow tcp seq issues (ooo, retx, lost) over ZMQ 2020-03-04 15:01:24 +01:00
Alfredo Cardigliano
c3d3984285 Add more info to flow serialize (json) 2020-02-26 13:23:07 +01:00
emanuele-f
15898e8dad Flow dissection code cleanup.
The Flow API is now composed of the following methods:
  - Flow::setDetectedProtocol to manually set a protocol on the flow
    and terminate the dissection
  - Flow::processPacket to run the nDPI dissection on the raw packets
  - Flow::endProtocolDissection to terminate or give up the dissection

Also fixes the flow sampling rate preference, which was never used.
2020-02-17 17:36:06 +01:00
Luca Deri
8786c9fb30 Added support for ALPN and Client Supported TLS Protocols reported by nDPI 2020-02-17 16:29:24 +01:00
emanuele-f
107c9e50f2 Add shadow dns query to prevent use after free 2020-02-14 11:01:05 +01:00
Simone Mainardi
f7d79e50f9 Implements anti stall when dumping flows (MySQL, nIndex, ES, ...)
Implements #3411
2020-02-13 11:36:48 +01:00
emanuele-f
7105cb8a65 Add score and user script config link into the flow details additional status 2020-02-07 20:02:19 +01:00
emanuele-f
e3d3d3992f Replace an existing flow alert if a more critical problem is found
Also add the flow score into the database
2020-02-07 19:20:57 +01:00
emanuele-f
0a0a3c4537 Rework flow status accounting 2020-02-07 19:17:07 +01:00
Simone Mainardi
38ee7f8439 Fixes for idle transitions not performed
Fixes #3394

Commit also affects #3106 as now DNS flows are adjusted
periodically when they contain more than one query
2020-02-07 19:13:45 +01:00
Luca Deri
f1d8fb00f4 Handled partial protocol detection 2020-02-06 22:01:32 +01:00
Simone Mainardi
05750cf3ef Fixes and cleanup 2020-02-05 18:07:54 +01:00
emanuele-f
8d51dae0e5 Implement Misbehaving vs Alerted flows chart 2020-01-27 15:37:36 +01:00
emanuele-f
566b9ece0b Score changes
- Move score from status definition to user scripts
- Separate flow score counter from the peers score
- Create a new HostScore class to hold the score data
2020-01-16 18:11:14 +01:00
emanuele-f
01d5d83f21 Account host score also on volatile flows (e.g. scans)
Such flows may go idle too early and miss the minute.lua iteration
2020-01-16 14:08:49 +01:00
emanuele-f
c336cca0e7 Improve host score calculation 2020-01-15 18:42:48 +01:00
emanuele-f
d7528e1628 Score improvements
The score is now calculated differently on the client and on the server of the flow.
The hosts flow is updated every minute and charted.
It's now possible to trigger an alert when the score threshold is exceeded
2020-01-15 12:34:16 +01:00
emanuele-f
93a96eb214 Code cleanup 2020-01-13 15:20:38 +01:00
Simone Mainardi
76391ff6a5 Resores flow alerts for view interfaces 2020-01-08 18:18:00 +01:00
Simone Mainardi
8cb1db967a Improvements to the probing traffic detection algorithm 2020-01-08 10:46:53 +01:00
Luca Deri
57e6a93065 Improved scheduling algorithm to guarantee minimum service time when number of entries allow
Updated (C)
2020-01-06 23:36:36 +01:00
Simone Mainardi
1b73a89dae Fixes detection of TCP connection-refused flow status 2020-01-03 12:37:41 +01:00
Simone Mainardi
417c4a3146 Reworks handling of TCP flags for both ZMQ and packet interfaces
This commit allows simplifies the code and also allows flags-based alerts
to be triggered for packet and non-packet interfaces

Fixes #3167
2020-01-02 19:21:44 +01:00
Simone Mainardi
0fc220bd0a Removes unused cli2srv_direction Flow class member 2020-01-02 15:04:49 +01:00
Luca
92ea3cf5a0 TLS certificate improvements 2020-01-02 09:36:31 +01:00
emanuele-f
87c336a4aa Reduce cpu usage due to the loading of flow.lua
The FlowAlertCheckLuaEngine is now cached into the (reused) vm, so it is reloaded
only when needed (e.g. a script on disk changes).
2019-12-31 18:27:14 +01:00
Simone Mainardi
2abb8cfde5 Reworks flow scripts deadlines 2019-12-27 20:50:53 +01:00
Luca Deri
c5fbb49bfd Fix for memory corruption in flow initialization 2019-12-26 23:00:45 +01:00
emanuele-f
8b7d711a36 Add option to disable flow user scripts invocation 2019-12-23 18:03:19 +01:00