Fix detection of WebDAV and Gnutella (over HTTP)
Fix detection of z3950
Add two fuzzers to test `ndpi_memmem()` and `ndpi_strnstr()`
Remove some dead code:
* RTP: the same exact check is performed at the very beginning of the
function
* MQTT: use a better helper to exclude the protocol
* Colletd: `ndpi_hostname_sni_set()` never fails
Update pl7m code (fix a Use-of-uninitialized-value error)
After a flow has been classified as RTP or RTCP, nDPI might analyse more
packets to look for STUN/DTLS packets, i.e. to try to tell if this flow
is a "pure" RTP/RTCP flow or if the RTP/RTCP packets are multiplexed with
STUN/DTLS.
Useful for proper (sub)classification when the beginning of the flows
are not captured or if there are lost packets in the the captured traffic.
Disabled by default
RFC4571 is not the only way to wrap RTP messages in TCP streams.
For example, when RTP is encapsulated over TURN flows (i.e. via DATA
attribute) there is no additional framing.
See also 6127e0490
There is some overlap between RTP and Raknet detection: give precedence
to RTP logic.
Consequences:
* Raknet might require a little bit more packets for some flows (not a
big issue)
* some very small (1-2 pkts) Raknet flows are not classified (not sure
what do do about that..)
Avoid code duplication between these two protocols.
We remove support for RTCP over TCP; it is quite rare to find this kind
of traffic and, more important, we have never had support for RTP
over TCP: we should try to add both detecion as follow-up.
Fix a message log in the LINE code
* Add Omron FINS protocol dissector
* Add a kludge to avoid invalid FINS over UDP detection as SkypeTeams and RTP
* Update unit test results
* Update protocols.rst
* Remove dummy flows from fins.pcap
1) Public API/headers in `src/include/` [as it has always been]
2) Private API/headers in `src/lib/`
Try to keep the "ndpi_" prefix only for the public functions
Try to have a faster classification, on first packet; use standard extra
dissection data path for sub-classification, metadata extraction and
monitoring.
STUN caches:
* use the proper confidence value
* lookup into the caches only once per flow, after having found a proper
STUN classification
Add identification of Telegram VoIP calls.
Some notes:
* libinjection: according to https://github.com/libinjection/libinjection/issues/44,
it seems NULL characters are valid in the input string;
* RTP: `rtp_get_stream_type()` is called only for RTP packets; if you
want to tell RTP from RTCP you should use `is_rtp_or_rtcp()`;
* TLS: unnecessary check; we already make the same check just above, at
the beginning of the `while` loop
The two fields `flow->flow_type` and `flow->protos.rtp.stream_type` are
pretty much identical: rename the former in `flow->flow_multimedia_type`
and remove the latter.
Try avoiding false positives: look for 3 RTP packets before classifing
the flow as such.
Add a generic function `is_rtp_or_rtcp()` to identify RTP/RTCP packets also
in other dissectors (see 3608ab01b commit message for an example)
The goal is to have Zoom flows classified as "Encrypted" and not as
"Cleartext".
Start documenting the list of protocols supported by nDPI;
format, verbosity and content are still a work-in-progress.
The checks `isValidMSRTPType(..) == 1` is a subset of
`is_valid_rtp_payload_type()` so this if-branch is never reached.
More importantly, the article describing how to detect Microsoft Lync and
Skype for Business is from 2014. These payload types are static or they
are in the dynamic range: in both cases, these values might be used (and
they are used indeed) pretty much by every application.
Bottom line: we can't use PT alone to identify a specific protocol.
Keep the list, since it is used to tell audio streams from video ones.
Remove `FUZZING_BUILD_MODE_UNSAFE_FOR_PRODUCTION` define from
`fuzz/Makefile.am`; it is already included by the main configure script
(when fuzzing).
Add a knob to force disabling of AESNI optimizations: this way we can
fuzz also no-aesni crypto code.
Move CRC32 algorithm into the library.
Add some fake traces to extend fuzzing coverage. Note that these traces
are hand-made (via scapy/curl) and must not be used as "proof" that the
dissectors are really able to identify this kind of traffic.
Some small updates to some dissectors:
CSGO: remove a wrong rule (never triggered, BTW). Any UDP packet starting
with "VS01" will be classified as STEAM (see steam.c around line 111).
Googling it, it seems right so.
XBOX: XBOX only analyses UDP flows while HTTP only TCP ones; therefore
that condition is false.
RTP, STUN: removed useless "break"s
Zattoo: `flow->zattoo_stage` is never set to any values greater or equal
to 5, so these checks are never true.
PPStream: `flow->l4.udp.ppstream_stage` is never read. Delete it.
TeamSpeak: we check for `flow->packet_counter == 3` just above, so the
following check `flow->packet_counter >= 3` is always false.
The application may enable only some protocols.
Disabling a protocol means:
*) don't register/use the protocol dissector code (if any)
*) disable classification by-port for such a protocol
*) disable string matchings for domains/certificates involving this protocol
*) disable subprotocol registration (if any)
This feature can be tested with `ndpiReader -B list_of_protocols_to_disable`.
Custom protocols are always enabled.
Technically speaking, this commit doesn't introduce any API/ABI
incompatibility. However, calling `ndpi_set_protocol_detection_bitmask2()`
is now mandatory, just after having called `ndpi_init_detection_module()`.
Most of the diffs (and all the diffs in `/src/lib/protocols/`) are due to
the removing of some function parameters.
Fix the low level macro `NDPI_LOG`. This issue hasn't been detected
sooner simply because almost all the code uses only the helpers `NDPI_LOG_*`
Load some custom configuration (like in the unit tests) and factorize some
(fuzzing) common code.
There is no way to pass file paths to the fuzzers as parameters. The safe
solution seems to be to load them from the process working dir. Anyway,
missing file is not a blocking error.
Remove some dead code (found looking at the coverage report)
Static analyzer complains about dereferencing `packet->udp` before
checking.
Since this function is called only with UDP flows, remove the check.
Close: #1792
These flows are classifed as `LINE_CALL`; another option was
`RTP/LINE_CALL`. No sure about the best solution...
Extend LINE domains list.
Remove RTP dead code.
As a general rule, the higher the confidence value, the higher the
"reliability/precision" of the classification.
In other words, this new field provides an hint about "how" the flow
classification has been obtained.
For example, the application may want to ignore classification "by-port"
(they are not real DPI classifications, after all) or give a second
glance at flows classified via LRU caches (because of false positives).
Setting only one value for the confidence field is a bit tricky: more
work is probably needed in the next future to tweak/fix/improve the logic.
There are no valid reasons for a (generic) protocol to ignore IPv6
traffic.
Note that:
* I have not found the specifications of "CheckPoint High Availability
Protocol", so I don't know how/if it supports IPv6
* all LRU caches are still IPv4 only
Even if src_id/dst_id stuff is probably useless (see #1279), the right
way to update the protocol classification is via `ndpi_set_detected_protocol()`
We can write to `flow->protos` only after a proper classification.
This issue has been found in Kerberos, DHCP, HTTP, STUN, IMO, FTP,
SMTP, IMAP and POP code.
There are two kinds of fixes:
* write to `flow->protos` only if a final protocol has been detected
* move protocol state out of `flow->protos`
The hard part is to find, for each protocol, the right tradeoff between
memory usage and code complexity.
Handle Kerberos like DNS: if we find a request, we set the protocol
and an extra callback to further parsing the reply.
For all the other protocols, move the state out of `flow->protos`. This
is an issue only for the FTP/MAIL stuff.
Add DHCP Class Identification value to the output of ndpiReader and to
the Jason serialization.
Extend code coverage of fuzz tests.
Close#1343Close#1342
There are no real reasons to embed `struct ndpi_packet_struct` (i.e. "packet")
in `struct ndpi_flow_struct` (i.e. "flow"). In other words, we can avoid
saving dissection information of "current packet" into the "flow" state,
i.e. in the flow management table.
The nDPI detection module processes only one packet at the time, so it is
safe to save packet dissection information in `struct ndpi_detection_module_struct`,
reusing always the same "packet" instance and saving a huge amount of memory.
Bottom line: we need only one copy of "packet" (for detection module),
not one for each "flow".
It is not clear how/why "packet" ended up in "flow" in the first place.
It has been there since the beginning of the GIT history, but in the original
OpenDPI code `struct ipoque_packet_struct` was embedded in
`struct ipoque_detection_module_struct`, i.e. there was the same exact
situation this commit wants to achieve.
Most of the changes in this PR are some boilerplate to update something
like "flow->packet" into something like "module->packet" throughout the code.
Some attention has been paid to update `ndpi_init_packet()` since we need
to reset some "packet" fields before starting to process another packet.
There has been one important change, though, in ndpi_detection_giveup().
Nothing changed for the applications/users, but this function can't access
"packet" anymore.
The reason is that this function can be called "asynchronously" with respect
to the data processing, i.e in context where there is no valid notion of
"current packet"; for example ndpiReader calls it after having processed all
the traffic, iterating the entire session table.
Mining LRU stuff seems a bit odd (even before this patch): probably we need
to rethink it, as a follow-up.
