CIFuzz (based on oss-fuzz) is the GitHub action/CI job that runs fuzz
targets on pull requests. It only runs the fuzzers affected by a pull
request/commit. Otherwise it will divide up the allotted fuzzing time
among all fuzzers in the project.
Since:
* we have more than 20 fuzzers and most of them use the custom memory
allocation functions (to force allocation failures) even if they are not
strictly about DPI stuff;
* we need to keep fuzzing time relatively small (to avoid waiting the CI
results for a long time)
it is important that fuzzers dependencies (which are based on *files*
changed by the single commit/PR) are as small as possible.
Bottom line: move all the low-level allocation callbacks to a dedicated
file; this way most of the fuzzers don't depend anymore on `ndpi_main.c`
file (which is touched by ever commit/PR).
The goal is to have only the "most important" fuzzers running during (most
of) the CI.
RFC 6066 3: "Literal IPv4 and IPv6 addresses are not permitted in
"HostName"."
Don't set this risk if we have a valid sub-classification (example:
via certificate)
Since a similar risk already exists for HTTP hostnames, reuse it, with a
more generic name.
As a side effect of this change, the following memory leak has been
fixed.
```
==55523==ERROR: LeakSanitizer: detected memory leaks
Direct leak of 2 byte(s) in 1 object(s) allocated from:
#0 0x55657d01b7be in malloc (/home/ivan/svnrepos/nDPI/fuzz/fuzz_ndpi_reader_alloc_fail+0x5b27be) (BuildId: 9dfa7bd9081421417b8b1c2e8b44633a9bfcfe52)
#1 0x55657d169182 in ndpi_malloc /home/ivan/svnrepos/nDPI/src/lib/ndpi_main.c:243:25
#2 0x55657d1694c8 in ndpi_strdup /home/ivan/svnrepos/nDPI/src/lib/ndpi_main.c:310:13
#3 0x55657d25ddc7 in processClientServerHello /home/ivan/svnrepos/nDPI/src/lib/protocols/tls.c:2308:46
#4 0x55657d2701b9 in processTLSBlock /home/ivan/svnrepos/nDPI/src/lib/protocols/tls.c:895:5
#5 0x55657d26a408 in ndpi_search_tls_udp /home/ivan/svnrepos/nDPI/src/lib/protocols/tls.c:1223:11
```
Found by oss-fuzz
See: https://bugs.chromium.org/p/oss-fuzz/issues/detail?id=56466
We can't write `flow->protos` union until we are really sure about protocol
classification
```
==28334==ERROR: AddressSanitizer: SEGV on unknown address (pc 0x558db5554512 bp 0x000000000000 sp 0x7ffcb22c2880 T0)
==28334==The signal is caused by a READ memory access.
==28334==Hint: this fault was caused by a dereference of a high value address (see register values below). Disassemble the provided pc to learn which register was used.
#0 0x558db5554512 in __asan::Allocator::Deallocate(void*, unsigned long, unsigned long, __sanitizer::BufferedStackTrace*, __asan::AllocType) (/home/ivan/svnrepos/nDPI/fuzz/fuzz_process_packet+0x48e512) (BuildId: 2f71e395637a7b748f36d5a04c7281f18b1128d7)
#1 0x558db55ea54b in __interceptor_free (/home/ivan/svnrepos/nDPI/fuzz/fuzz_process_packet+0x52454b) (BuildId: 2f71e395637a7b748f36d5a04c7281f18b1128d7)
#2 0x558db56977ca in ndpi_free /home/ivan/svnrepos/nDPI/src/lib/ndpi_main.c:274:7
#3 0x558db56c20e3 in ndpi_free_flow_data /home/ivan/svnrepos/nDPI/src/lib/ndpi_main.c:5175:2
#4 0x558db569783f in ndpi_free_flow /home/ivan/svnrepos/nDPI/src/lib/ndpi_main.c:8394:5
#5 0x558db5627936 in LLVMFuzzerTestOneInput /home/ivan/svnrepos/nDPI/fuzz/fuzz_process_packet.c:38:3
```
Found by oss-fuzz
See: https://bugs.chromium.org/p/oss-fuzz/issues/detail?id=56272
Latest RoaringBitmap version (introduced with bf413afb) triggers a new
warning with GCC-7:
```
ivan@ivan-Latitude-E6540:~/svnrepos/nDPI(dev)$ CC=gcc-7 CXX=g++-7 ./autogen.sh && make -s
autoreconf: Entering directory `.'
[...]
third_party/src/roaring.c:1815:1: warning: ‘no_sanitize’ attribute directive ignored [-Wattributes]
static inline int array_container_cardinality(const array_container_t *array) {
^~~~~~
third_party/src/roaring.c:1964:5: warning: ‘no_sanitize’ attribute directive ignored [-Wattributes]
const array_container_t *container2) {
[..]
```
The core issue is that `no_sanitize` attribute is defined only for GCC
>= 8.
That breaks the CI since we still use GCC-7 and `-Werror`: add a simple
workaround.
Fix compilation on Windows
Remove `FUZZING_BUILD_MODE_UNSAFE_FOR_PRODUCTION` define from
`fuzz/Makefile.am`; it is already included by the main configure script
(when fuzzing).
Add a knob to force disabling of AESNI optimizations: this way we can
fuzz also no-aesni crypto code.
Move CRC32 algorithm into the library.
Add some fake traces to extend fuzzing coverage. Note that these traces
are hand-made (via scapy/curl) and must not be used as "proof" that the
dissectors are really able to identify this kind of traffic.
Some small updates to some dissectors:
CSGO: remove a wrong rule (never triggered, BTW). Any UDP packet starting
with "VS01" will be classified as STEAM (see steam.c around line 111).
Googling it, it seems right so.
XBOX: XBOX only analyses UDP flows while HTTP only TCP ones; therefore
that condition is false.
RTP, STUN: removed useless "break"s
Zattoo: `flow->zattoo_stage` is never set to any values greater or equal
to 5, so these checks are never true.
PPStream: `flow->l4.udp.ppstream_stage` is never read. Delete it.
TeamSpeak: we check for `flow->packet_counter == 3` just above, so the
following check `flow->packet_counter >= 3` is always false.
Autodetecting the needed buffer size is quite complex (especially with
float/double values) so it is mandatory to properly check for
`ndpi_snprintf` truncation.
These issues have been undetected so far probably because the default
buffer is big enough for all common cases.
Add an example of usage of `ndpi_deserialize_clone_all()` (taken from
`ntopng`)
ubuntu-latest s390x CI doesn't like snapshot bigger than 262144 bytes.
Fix an error found by fuzz CI
```
=================================================================
==55399==ERROR: AddressSanitizer: SEGV on unknown address 0x000000000008 (pc 0x561e41cb684d bp 0x7ffd54ce3650 sp 0x7ffd54ce3520 T0)
==55399==The signal is caused by a READ memory access.
==55399==Hint: address points to the zero page.
#0 0x561e41cb684d in ndpi_network_ptree_match /home/ivan/svnrepos/nDPI/src/lib/ndpi_main.c:2321:41
#1 0x561e41d30879 in ndpi_guess_undetected_protocol /home/ivan/svnrepos/nDPI/src/lib/ndpi_main.c:7767:8
#2 0x561e41ca804d in LLVMFuzzerTestOneInput /home/ivan/svnrepos/nDPI/fuzz/fuzz_config.cpp:104:5
#3 0x561e41bb96a0 in fuzzer::Fuzzer::ExecuteCallback(unsigned char const*, unsigned long) (/home/ivan/svnrepos/nDPI/fuzz/fuzz_config+0x4726a0) (BuildId: d4741c753aafe7c0df2681a592b7df16b38240e9)
#4 0x561e41ba3c2f in fuzzer::RunOneTest(fuzzer::Fuzzer*, char const*, unsigned long) (/home/ivan/svnrepos/nDPI/fuzz/fuzz_config+0x45cc2f) (BuildId: d4741c753aafe7c0df2681a592b7df16b38240e9)
#5 0x561e41ba96f6 in fuzzer::FuzzerDriver(int*, char***, int (*)(unsigned char const*, unsigned long)) (/home/ivan/svnrepos/nDPI/fuzz/fuzz_config+0x4626f6) (BuildId: d4741c753aafe7c0df2681a592b7df16b38240e9)
#6 0x561e41bd22e2 in main (/home/ivan/svnrepos/nDPI/fuzz/fuzz_config+0x48b2e2) (BuildId: d4741c753aafe7c0df2681a592b7df16b38240e9)
#7 0x7f94f0e5c082 in __libc_start_main /build/glibc-SzIz7B/glibc-2.31/csu/../csu/libc-start.c:308:16
#8 0x561e41b9eb0d in _start (/home/ivan/svnrepos/nDPI/fuzz/fuzz_config+0x457b0d) (BuildId: d4741c753aafe7c0df2681a592b7df16b38240e9)
```
