Commit graph

658 commits

Author SHA1 Message Date
Luca Deri
35ef56cc24 Crash fix when -f is specified with a non-existing pcap file (-i) 2024-11-04 22:14:09 +01:00
Luca Deri
183175fc6b Exported is_ndpi_proto definition 2024-11-04 22:02:13 +01:00
Luca Deri
412ca8700f Added HTTP credentials extraction 2024-10-31 21:20:46 +01:00
Ivan Nardi
9da99075aa
TLS: export heuristic fingerprint as metadata (#2609) 2024-10-28 23:36:51 +01:00
Ivan Nardi
a55d8d95bc
ndpiReader: fix command line options used by wireshark (#2605) 2024-10-21 18:26:28 +02:00
Ivan Nardi
51556113a3
ndpiReader: add some statistics about monitoring (#2602) 2024-10-19 19:44:00 +02:00
Luca Deri
afc4d9e34d Added -L <domain suffix> for loading domain suffixes
Exported domainanme in JSON file (-K JSON)
2024-10-15 22:25:48 +02:00
Luca Deri
6b6dad4fdb Implemented nDPI TCP fingerprint 2024-10-15 21:21:05 +02:00
Ivan Nardi
521d0ca7a0
Add monitoring capability (#2588)
Allow nDPI to process the entire flows and not only the first N packets.
Usefull when the application is interested in some metadata spanning the
entire life of the session.

As initial step, only STUN flows can be put in monitoring.

See `doc/monitoring.md` for further details.

This feature is disabled by default.

Close #2583
2024-10-14 18:05:35 +02:00
Luca Deri
faaa5c5799 Added support for printing JA4r when enabled 2024-10-11 19:24:06 +02:00
Ivan Nardi
3e74c95d19
Fix unit tests on CI on GitHub Actions (#2587)
On CI, tests run in parallel, because of `NDPI_FORCE_PARALLEL_UTESTS`
define
2024-10-11 13:08:48 +02:00
Luca Deri
2e5edd2cc9 Added -N option for dumping/restoring the DNS cache (when enabled)
Example ndpiReader -i en0 --cfg=dpi.address_cache_size,32768 -N /tmp/a
2024-10-10 18:10:03 +02:00
Luca Deri
b9348e9d6e Added new API calls for serializing/restoring the DNS cache
-  bool ndpi_address_cache_dump(struct ndpi_address_cache *cache, char *path, u_int32_t epoch_now);
-  u_int32_t ndpi_address_cache_restore(struct ndpi_address_cache *cache, char *path, u_int32_t epoch_now);
2024-10-10 16:51:45 +02:00
Luca Deri
89316ab4c0 Test unit fix 2024-10-08 08:58:24 +02:00
Luca Deri
55fa92490a Implemented (disabled by default) DNS host cache. You can set the cache size as follows:
ndpiReader --cfg=dpi.address_cache_size,1000 -i <pcap>.pcap

In the above example the cache has up to 1000 entries.

In jcase ndpiReader exports data in JSON, the cache hostname (if found) is exported in the field server_hostname
2024-10-07 20:08:53 +02:00
Ivan Nardi
8972b74fd0
Fix builds on Windows (#2580)
Quick fix with latest Windows image on GitHub CI, where we got:
```
 ndpiReader.c:2860:38: error: '%s' directive output may be truncated writing up to 64 bytes into a region of size 63 [-Werror=format-truncation=]
 2860 |     snprintf(srcip, sizeof(srcip), "[%s]", flow->src_name);
      |                                      ^~
ndpiReader.c:2860:5: note: 'snprintf' output between 3 and 67 bytes into a destination of size 64
 2860 |     snprintf(srcip, sizeof(srcip), "[%s]", flow->src_name);
      |     ^~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
ndpiReader.c:2861:38: error: '%s' directive output may be truncated writing up to 64 bytes into a region of size 63 [-Werror=format-truncation=]
 2861 |     snprintf(dstip, sizeof(dstip), "[%s]", flow->dst_name);
      |                                      ^~
ndpiReader.c:2861:5: note: 'snprintf' output between 3 and 67 bytes into a destination of size 64
 2861 |     snprintf(dstip, sizeof(dstip), "[%s]", flow->dst_name);
      |     ^~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
```
2024-10-01 15:51:16 +02:00
Ivan Nardi
bfab402e05
wireshark: extcap: allow configuration of OpenVPN/TLS heuristics via GUI (#2576) 2024-09-30 16:55:22 +02:00
Ivan Nardi
e2ed23a72a
Let the library returning the packet direction calculated internally (#2572)
wireshark, lua: add basic analysis of possible obfuscated flows
2024-09-27 18:51:47 +02:00
Ivan Nardi
ddd08f913c
Add some heuristics to detect encrypted/obfuscated/proxied TLS flows (#2553)
Based on the paper: "Fingerprinting Obfuscated Proxy Traffic with
Encapsulated TLS Handshakes".
See: https://www.usenix.org/conference/usenixsecurity24/presentation/xue-fingerprinting

Basic idea:
* the packets/bytes distribution of a TLS handshake is quite unique
* this fingerprint is still detectable if the handshake is
encrypted/proxied/obfuscated

All heuristics are disabled by default.
2024-09-24 14:20:31 +02:00
Luca Deri
9a8dc640be buffer lenghtt is now returned by ndpi_quick_encrypt() and ndpi_quick_deecrypt() 2024-09-24 12:51:55 +02:00
Luca Deri
9f3b7cfd65
Added ndpi_quick_encrypt() ndpi_quick_decrypt() APi calls (#2568)
* Added ndpi_quick_encrypt() ndpi_quick_decrypt(0 APi calls based on AES

* Added aes.c
2024-09-24 09:40:21 +02:00
Luca Deri
191694f797 Implemented ndpi_strrstr()
Fixed bug in ndpi_get_host_domain
2024-09-19 13:18:26 +02:00
Luca
1d1edfc1c7 domain lists are not loaded when -E is used 2024-09-17 19:04:01 +02:00
Ivan Nardi
1197cbdf58
wireshark: extcap: rework trailer header (#2557) 2024-09-16 22:19:20 +02:00
Luca
6de91c7895 Reworked fingerprint export now in JSON 2024-09-16 09:21:17 +02:00
Luca Deri
fda3730cf0 Added -E option for dumping flow fingerprint 2024-09-15 19:38:33 +02:00
Luca Deri
7fdc4b2472
Implemented algorithms for K-Nearest Neighbor Search (KNN) (#2554)
* Extended API with functions for vector similarity based on KD-trees https://en.wikipedia.org/wiki/K-d_tree

ndpi_kd_tree* ndpi_kd_create(u_int num_dimensions);
void ndpi_kd_free(ndpi_kd_tree *tree);
void ndpi_kd_clear(ndpi_kd_tree *tree);
bool ndpi_kd_insert(ndpi_kd_tree *tree, const double *data_vector, void *user_data);
ndpi_kd_tree_result *ndpi_kd_nearest(ndpi_kd_tree *tree, const double *data_vector);
u_int32_t ndpi_kd_num_results(ndpi_kd_tree_result *res);
bool ndpi_kd_result_end(ndpi_kd_tree_result *res);
double* ndpi_kd_result_get_item(ndpi_kd_tree_result *res, double **user_data);
bool ndpi_kd_result_next(ndpi_kd_tree_result *res);
void ndpi_kd_result_free(ndpi_kd_tree_result *res);
double ndpi_kd_distance(double *a1, double *b2, u_int num_dimensions);
2024-09-10 16:22:06 +02:00
Nardi Ivan
23ae3d0c26 wireshark: extcap: export flow risk info 2024-09-05 16:27:24 +02:00
Nardi Ivan
e562cdc5bd wireshark: extcap: fix output data link type
Avoid forcing `DLT_EN10MB` but use the same data link type of the input
pcap.
This way, we can use extcap functionality with input traces having Linux
"cooked" capture encapsulation, i.e. traces captured on "any" interface
2024-09-05 16:27:24 +02:00
Luca Deri
d769b23e05 Added print_ndpi_address_port in nDPi API 2024-08-27 16:38:35 +02:00
Vladimir Gavrilov
aec2e2fbb8
Endian-independent implementation of IEEE 802.3 CRC32 (#2529) 2024-08-25 11:54:17 +02:00
Luca Deri
53a6bae365 Introduced ndpi_master_app_protocol typedef 2024-08-24 16:30:58 +02:00
Luca Deri
9b1736aa89 Initialization fix 2024-08-22 12:26:17 +02:00
Luca Deri
e72255445c Fixed initialization 2024-08-22 12:08:12 +02:00
Ivan Nardi
2740a4f4e3
Update all IP lists (#2515)
The `suffix_id` is simply an incremental index (see
`ndpi_load_domain_suffixes`), so its value might changes every time we
update the public suffix list.
2024-08-02 15:06:08 +02:00
Ivan Nardi
65e31b0ea3
FPC: small improvements (#2512)
Add printing of fpc_dns statistics and add a general cconfiguration option.
Rework the code to be more generic and ready to handle other logics.
2024-07-22 17:42:23 +02:00
Petr
92d0b8d91f
ndpi_strncasestr: optimization, fixes, tests (#2507) 2024-07-18 19:40:09 +02:00
Petr
2f66a6a3e1
ndpi_memmem: optimized, fixed bug, added tests (#2499) 2024-07-15 08:35:10 +02:00
Petr
e059daa0f1
Optimize performance of ndpi_strnstr() and possible bugfix (#2494) 2024-07-15 08:34:08 +02:00
Ivan Nardi
843e487270
Add infrastructure for explicit support of Fist Packet Classification (#2488)
Let's start with some basic helpers and with FPC based on flow addresses.

See: #2322
2024-07-03 18:02:07 +02:00
Luca Deri
731b75b44c Modified separator from , (comma) to | (pipe) as some fields such as the HTTP user agent as sometimes they contain commas and create parsing problems 2024-07-01 09:53:38 +02:00
Nardi Ivan
556f892a56 wireshark: lua: export some metadata
Export some metadata (for the moment, SNI and TLS fingerprints) to
Wireshark/tshark via extcap.
Note that:
* metadata are exported only once per flow
* metadata are exported (all together) when nDPI stopped processing
the flow

Still room for a lot of improvements!
In particular:
* we need to add some boundary checks (if we are going to export other
attributes)
* we should try to have a variable length trailer
2024-06-25 16:39:45 +02:00
Nardi Ivan
b5afa165f0 wireshark: extcap: restore filtering mechanism 2024-06-25 16:39:45 +02:00
Mark Jeffery
aa1d7247d1
Added default port mappings to ndpiReader help -H (#2477)
Close #2125
2024-06-19 13:47:18 +02:00
Nardi Ivan
526cf6f291 Zoom: remove "stun_zoom" LRU cache
Since 070a0908b we are able to detect P2P calls directly from the packet
content, without any correlation among flows
2024-06-17 10:19:55 +02:00
Mark Jeffery
312dc424bd
Added NDPI_PROTOCOL_NTOP assert and removed percentage comparison (#2460)
Close #2413
2024-06-10 19:45:19 +02:00
Ivan Nardi
0109014f2c
Follow-up of 2093ac5bf (#2451) 2024-05-21 12:47:25 +02:00
Luca Deri
2093ac5bf6 Minor dissector optimizations 2024-05-20 12:17:04 +02:00
Luca Deri
42dba2e4af Added dpi.compute_entropy configuration parameter 2024-05-18 09:46:15 +02:00
Ivan Nardi
a064261e85
Revert ndpi_strnstr() optimization introduced in a813121e0 (#2439)
New implementation fails tests 11b, 12 and 13.
Revert to the original (BSD) implementation (with also some basic
parameters check)
2024-05-11 23:37:31 +02:00