vrr/nDPI
2
0
Fork 0
mirror of https://github.com/vel21ripn/nDPI.git synced 2026-05-01 16:30:17 +00:00
Code Issues Projects Releases Packages Wiki Activity Actions 5

Add the concept of protocols stack: more than 2 protocols per flow (#2913)

Browse source 
The idea is to remove the limitation of only two protocols ("master" and
"app") in the flow classifcation.
This is quite handy expecially for STUN flows and, in general, for any
flows where there is some kind of transitionf from a cleartext protocol
to TLS: HTTP_PROXY -> TLS/Youtube; SMTP -> SMTPS (via STARTTLS msg).

In the vast majority of the cases, the protocol stack is simply
Master/Application.

Examples of real stacks (from the unit tests)  different from the standard
"master/app":
* "STUN.WhatsAppCall.SRTP": a WA call
* "STUN.DTLS.GoogleCall": a Meet call
* "Telegram.STUN.DTLS.TelegramVoip": a Telegram call
* "SMTP.SMTPS.Google": a SMTP connection to Google server started in
  cleartext and updated to TLS
* "HTTP.Google.ntop": a HTTP connection to a Google domain (match via
  "Host" header) and to a ntop server (match via "Server" header)

The logic to create the stack is still a bit coarse: we have a decade of
code try to push everything in only ywo protocols... Therefore, the
content of the stack is still **highly experimental** and might change
in the next future; do you have any suggestions?

It is quite likely that the legacy fields "master_protocol" and
"app_protocol" will be there for a long time.

Add some helper to use the stack:
```
ndpi_stack_get_upper_proto();
ndpi_stack_get_lower_proto();
bool ndpi_stack_contains(struct ndpi_proto_stack *s, u_int16_t proto_id);
bool ndpi_stack_is_tls_like(struct ndpi_proto_stack *s);
bool ndpi_stack_is_http_like(struct ndpi_proto_stack *s);

```

Be sure new stack logic is compatible with legacy code:
```
assert(ndpi_stack_get_upper_proto(&flow->detected_protocol.protocol_stack) ==
       ndpi_get_upper_proto(flow->detected_protocol));
assert(ndpi_stack_get_lower_proto(&flow->detected_protocol.protocol_stack) ==
       ndpi_get_lower_proto(flow->detected_protocol));
```
This commit is contained in:
Ivan Nardi 2025-08-01 10:05:50 +02:00 • committed by GitHub
parent 79f0cbd32a
commit 8dd2220116
No known key found for this signature in database
GPG key ID: B5690EEEBB952194
689 changed files with 13984 additions and 13680 deletions
Download patch file Download diff file Expand all files Collapse all files

16
tests/cfgs/default/result/false_positives.pcapng.out
View file

@ -42,14 +42,14 @@ JA Host Stats:
1 91.238.181.21 1
1 TCP 10.17.24.50:4343 <-> 20.1.35.76:25 [proto: 3/SMTP][IP: 276/Azure][ClearText][Confidence: DPI][FPC: 276/Azure, Confidence: IP address][DPI packets: 7][cat: Email/3][Breed: Acceptable][21 pkts/18740 bytes <-> 12 pkts/1390 bytes][Goodput ratio: 93/43][< 1 sec][Hostname/SNI: server-1402abab.example.int][bytes ratio: 0.862 (Upload)][IAT c2s/s2c min/avg/max/stddev: 0/0 0/0 0/0 0/0][Pkt Len c2s/s2c min/avg/max/stddev: 66/66 892/116 1514/353 705/76][TCP Fingerprint: 2_32_5792_13ad4065e152/Unknown][PLAIN TEXT (220 server)][Plen Bins: 8,33,4,0,0,0,0,0,4,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,4,0,0,0,0,45,0,0]
2 UDP 10.192.92.81:52070 <-> 10.136.43.69:21048 [VLAN: 20][proto: 87/RTP][IP: 0/Unknown][Stream Content: Audio][Payload Type: ITU-T G.711 PCMA (8.0) / ITU-T G.711 PCMA (8.0)][ClearText][Confidence: DPI][FPC: 0/Unknown, Confidence: Unknown][DPI packets: 4][cat: Media/1][Breed: Acceptable][15 pkts/3330 bytes <-> 15 pkts/3330 bytes][Goodput ratio: 77/77][0.30 sec][bytes ratio: 0.000 (Mixed)][IAT c2s/s2c min/avg/max/stddev: 19/19 19/19 20/20 0/0][Pkt Len c2s/s2c min/avg/max/stddev: 222/222 222/222 222/222 0/0][PLAIN TEXT (UUUUUUUUUUUUU)][Plen Bins: 0,0,0,0,0,100,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0]
3 UDP 10.126.70.67:23784 <-> 10.236.7.225:50160 [VLAN: 107][proto: 87/RTP][IP: 0/Unknown][Stream Content: Audio][Payload Type: ITU-T G.711 PCMA (8.0) / ITU-T G.711 PCMA (8.0)][ClearText][Confidence: DPI][FPC: 0/Unknown, Confidence: Unknown][DPI packets: 7][cat: Media/1][Breed: Acceptable][18 pkts/3924 bytes <-> 12 pkts/2616 bytes][Goodput ratio: 79/79][0.34 sec][bytes ratio: 0.200 (Upload)][IAT c2s/s2c min/avg/max/stddev: 19/19 20/20 20/20 0/0][Pkt Len c2s/s2c min/avg/max/stddev: 218/218 218/218 218/218 0/0][PLAIN TEXT (UUUUUUUUU)][Plen Bins: 0,0,0,0,0,100,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0]
4 UDP 10.102.45.249:31046 <-> 10.133.48.100:21176 [VLAN: 10][proto: GTP:87/RTP][IP: 0/Unknown][Payload Type: Unknown (102.0) / Unknown (102.0)][ClearText][Confidence: DPI][FPC: 0/Unknown, Confidence: Unknown][DPI packets: 17][cat: Media/1][Breed: Acceptable][22 pkts/2860 bytes <-> 8 pkts/989 bytes][Goodput ratio: 34/30][0.44 sec][bytes ratio: 0.486 (Upload)][IAT c2s/s2c min/avg/max/stddev: 0/19 22/19 44/20 15/0][Pkt Len c2s/s2c min/avg/max/stddev: 130/113 130/124 130/130 0/8][Plen Bins: 10,90,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0]
5 UDP 10.133.32.101:36408 -> 10.110.31.25:1272 [VLAN: 10][proto: GTP:87/RTP][IP: 0/Unknown][Payload Type: AMR (118.0)][ClearText][Confidence: DPI][FPC: 0/Unknown, Confidence: Unknown][DPI packets: 20][cat: Media/1][Breed: Acceptable][20 pkts/2260 bytes -> 0 pkts/0 bytes][Goodput ratio: 24/0][0.38 sec][bytes ratio: 1.000 (Upload)][IAT c2s/s2c min/avg/max/stddev: 19/0 20/0 21/0 1/0][Pkt Len c2s/s2c min/avg/max/stddev: 113/0 113/0 113/0 0/0][Plen Bins: 100,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0]
6 TCP 91.238.181.21:35888 <-> 89.31.79.12:3389 [VLAN: 77][proto: 91.88/TLS.RDP][IP: 0/Unknown][Encrypted][Confidence: DPI][FPC: 0/Unknown, Confidence: Unknown][DPI packets: 5][cat: RemoteAccess/12][Breed: Acceptable][3 pkts/239 bytes <-> 2 pkts/1332 bytes][Goodput ratio: 20/91][0.07 sec][Risk: ** TLS (probably) Not Carrying HTTPS **** Missing SNI TLS Extn **** Desktop/File Sharing **** TLS Susp Extn **** Non-Printable/Invalid Chars Detected **** Possible Exploit Attempt **][Risk Score: 420][Risk Info: Invalid chars found in SNI: exploit or misconfiguration? / xsen??????????????????tsp / Extn id 9216 / Found RDP / SNI should a][TCP Fingerprint: 194_128_8192_6bb88f5575fd/Unknown][TLS (0589)][JA4: t00i001700_e3b0c44298fc_6d0650a004ef][PLAIN TEXT (Cookie)][Plen Bins: 33,33,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,33,0,0,0,0,0,0,0,0,0,0]
7 TCP 10.140.231.26:61202 <-> 159.65.12.169:443 [VLAN: 113][proto: GTP:7.251/HTTP.WebSocket][IP: 442/DigitalOcean][ClearText][Confidence: DPI][FPC: 442/DigitalOcean, Confidence: IP address][DPI packets: 4][cat: Web/5][Breed: Acceptable][2 pkts/557 bytes <-> 2 pkts/416 bytes][Goodput ratio: 58/45][0.20 sec][Hostname/SNI: wludo.superkinglabs.com][URL: wludo.superkinglabs.com:443/ws][StatusCode: 101][Server: nginx/1.12.2][Risk: ** Known Proto on Non Std Port **** HTTP Susp User-Agent **** HTTP Obsolete Server **][Risk Score: 200][Risk Info: Obsolete nginx server 1.12.2 / Empty or missing User-Agent / Expected on port 80][TCP Fingerprint: 2_64_65535_15db81ff8b0d/Unknown][PLAIN TEXT (GET /ws HTTP/1.1)][Plen Bins: 0,0,0,0,0,50,0,0,0,0,50,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0]
1 TCP 10.17.24.50:4343 <-> 20.1.35.76:25 [proto: 3/SMTP][Stack: SMTP][IP: 276/Azure][ClearText][Confidence: DPI][FPC: 276/Azure, Confidence: IP address][DPI packets: 7][cat: Email/3][Breed: Acceptable][21 pkts/18740 bytes <-> 12 pkts/1390 bytes][Goodput ratio: 93/43][< 1 sec][Hostname/SNI: server-1402abab.example.int][bytes ratio: 0.862 (Upload)][IAT c2s/s2c min/avg/max/stddev: 0/0 0/0 0/0 0/0][Pkt Len c2s/s2c min/avg/max/stddev: 66/66 892/116 1514/353 705/76][TCP Fingerprint: 2_32_5792_13ad4065e152/Unknown][PLAIN TEXT (220 server)][Plen Bins: 8,33,4,0,0,0,0,0,4,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,4,0,0,0,0,45,0,0]
2 UDP 10.192.92.81:52070 <-> 10.136.43.69:21048 [VLAN: 20][proto: 87/RTP][Stack: RTP][IP: 0/Unknown][Stream Content: Audio][Payload Type: ITU-T G.711 PCMA (8.0) / ITU-T G.711 PCMA (8.0)][ClearText][Confidence: DPI][FPC: 0/Unknown, Confidence: Unknown][DPI packets: 4][cat: Media/1][Breed: Acceptable][15 pkts/3330 bytes <-> 15 pkts/3330 bytes][Goodput ratio: 77/77][0.30 sec][bytes ratio: 0.000 (Mixed)][IAT c2s/s2c min/avg/max/stddev: 19/19 19/19 20/20 0/0][Pkt Len c2s/s2c min/avg/max/stddev: 222/222 222/222 222/222 0/0][PLAIN TEXT (UUUUUUUUUUUUU)][Plen Bins: 0,0,0,0,0,100,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0]
3 UDP 10.126.70.67:23784 <-> 10.236.7.225:50160 [VLAN: 107][proto: 87/RTP][Stack: RTP][IP: 0/Unknown][Stream Content: Audio][Payload Type: ITU-T G.711 PCMA (8.0) / ITU-T G.711 PCMA (8.0)][ClearText][Confidence: DPI][FPC: 0/Unknown, Confidence: Unknown][DPI packets: 7][cat: Media/1][Breed: Acceptable][18 pkts/3924 bytes <-> 12 pkts/2616 bytes][Goodput ratio: 79/79][0.34 sec][bytes ratio: 0.200 (Upload)][IAT c2s/s2c min/avg/max/stddev: 19/19 20/20 20/20 0/0][Pkt Len c2s/s2c min/avg/max/stddev: 218/218 218/218 218/218 0/0][PLAIN TEXT (UUUUUUUUU)][Plen Bins: 0,0,0,0,0,100,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0]
4 UDP 10.102.45.249:31046 <-> 10.133.48.100:21176 [VLAN: 10][proto: GTP:87/RTP][Stack: RTP][IP: 0/Unknown][Payload Type: Unknown (102.0) / Unknown (102.0)][ClearText][Confidence: DPI][FPC: 0/Unknown, Confidence: Unknown][DPI packets: 17][cat: Media/1][Breed: Acceptable][22 pkts/2860 bytes <-> 8 pkts/989 bytes][Goodput ratio: 34/30][0.44 sec][bytes ratio: 0.486 (Upload)][IAT c2s/s2c min/avg/max/stddev: 0/19 22/19 44/20 15/0][Pkt Len c2s/s2c min/avg/max/stddev: 130/113 130/124 130/130 0/8][Plen Bins: 10,90,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0]
5 UDP 10.133.32.101:36408 -> 10.110.31.25:1272 [VLAN: 10][proto: GTP:87/RTP][Stack: RTP][IP: 0/Unknown][Payload Type: AMR (118.0)][ClearText][Confidence: DPI][FPC: 0/Unknown, Confidence: Unknown][DPI packets: 20][cat: Media/1][Breed: Acceptable][20 pkts/2260 bytes -> 0 pkts/0 bytes][Goodput ratio: 24/0][0.38 sec][bytes ratio: 1.000 (Upload)][IAT c2s/s2c min/avg/max/stddev: 19/0 20/0 21/0 1/0][Pkt Len c2s/s2c min/avg/max/stddev: 113/0 113/0 113/0 0/0][Plen Bins: 100,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0]
6 TCP 91.238.181.21:35888 <-> 89.31.79.12:3389 [VLAN: 77][proto: 91.88/TLS.RDP][Stack: RDP.TLS][IP: 0/Unknown][Encrypted][Confidence: DPI][FPC: 0/Unknown, Confidence: Unknown][DPI packets: 5][cat: RemoteAccess/12][Breed: Acceptable][3 pkts/239 bytes <-> 2 pkts/1332 bytes][Goodput ratio: 20/91][0.07 sec][Risk: ** TLS (probably) Not Carrying HTTPS **** Missing SNI TLS Extn **** Desktop/File Sharing **** TLS Susp Extn **** Non-Printable/Invalid Chars Detected **** Possible Exploit Attempt **][Risk Score: 420][Risk Info: Invalid chars found in SNI: exploit or misconfiguration? / xsen??????????????????tsp / Extn id 9216 / Found RDP / SNI should a][TCP Fingerprint: 194_128_8192_6bb88f5575fd/Unknown][TLS (0589)][JA4: t00i001700_e3b0c44298fc_6d0650a004ef][PLAIN TEXT (Cookie)][Plen Bins: 33,33,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,33,0,0,0,0,0,0,0,0,0,0]
7 TCP 10.140.231.26:61202 <-> 159.65.12.169:443 [VLAN: 113][proto: GTP:7.251/HTTP.WebSocket][Stack: HTTP.WebSocket][IP: 442/DigitalOcean][ClearText][Confidence: DPI][FPC: 442/DigitalOcean, Confidence: IP address][DPI packets: 4][cat: Web/5][Breed: Acceptable][2 pkts/557 bytes <-> 2 pkts/416 bytes][Goodput ratio: 58/45][0.20 sec][Hostname/SNI: wludo.superkinglabs.com][URL: wludo.superkinglabs.com:443/ws][StatusCode: 101][Server: nginx/1.12.2][Risk: ** Known Proto on Non Std Port **** HTTP Susp User-Agent **** HTTP Obsolete Server **][Risk Score: 200][Risk Info: Obsolete nginx server 1.12.2 / Empty or missing User-Agent / Expected on port 80][TCP Fingerprint: 2_64_65535_15db81ff8b0d/Unknown][PLAIN TEXT (GET /ws HTTP/1.1)][Plen Bins: 0,0,0,0,0,50,0,0,0,0,50,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0]
Undetected flows:
1 UDP 192.168.12.156:37649 <-> 57.128.172.97:9981 [proto: 0/Unknown][IP: 0/Unknown][ClearText][Confidence: Unknown][FPC: 0/Unknown, Confidence: Unknown][DPI packets: 6][Breed: Unrated][3 pkts/230 bytes <-> 3 pkts/230 bytes][Goodput ratio: 45/45][1.03 sec][bytes ratio: 0.000 (Mixed)][IAT c2s/s2c min/avg/max/stddev: 498/498 505/504 512/511 7/6][Pkt Len c2s/s2c min/avg/max/stddev: 72/72 77/77 82/82 4/4][Risk: ** Susp Entropy **][Risk Score: 10][Risk Info: Entropy: 4.970 (Executable?)][Plen Bins: 33,66,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0]
1 UDP 192.168.12.156:37649 <-> 57.128.172.97:9981 [proto: 0/Unknown][Stack: Unknown][IP: 0/Unknown][ClearText][Confidence: Unknown][FPC: 0/Unknown, Confidence: Unknown][DPI packets: 6][Breed: Unrated][3 pkts/230 bytes <-> 3 pkts/230 bytes][Goodput ratio: 45/45][1.03 sec][bytes ratio: 0.000 (Mixed)][IAT c2s/s2c min/avg/max/stddev: 498/498 505/504 512/511 7/6][Pkt Len c2s/s2c min/avg/max/stddev: 72/72 77/77 82/82 4/4][Risk: ** Susp Entropy **][Risk Score: 10][Risk Info: Entropy: 4.970 (Executable?)][Plen Bins: 33,66,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0]