nDPI/ndpi-netfilter/INSTALL

-----------------
1. Kernel patches
-----------------

This version works without patches for the kernel.
The minimum required kernel version is 4.4.
If you need to use the extension 'connlabel' you need
to use a patch for the kernel.

Linux kernels 5.18 and higher that do not support
CONFIG_LIVEPATCH require a patch.

--------------------------------------------------------
2. The minimum requirements for the kernel configuration
--------------------------------------------------------

Kernel version >= 5.18 with "Kernel Live Patching"

  Processor type and features  --->
    [*] Kernel Live Patching
  Networking support  --->
    Networking options  --->
    [*] Network packet filtering framework (Netfilter)  --->
        Core Netfilter Configuration  --->
          Netfilter connection tracking support (M/Y)
	  "connlabel" match support (M)

  or

    CONFIG_LIVEPATCH=y
    CONFIG_NF_CONNTRACK=m or y
    CONFIG_NF_CONNTRACK_LABELS=y
    CONFIG_NETFILTER_XT_MATCH_CONNLABEL=m

Kernel version >= 5.18 with patch, without "Kernel Live Patching"

  Networking support  --->
    Networking options  --->
    [*] Network packet filtering framework (Netfilter)  --->
        Core Netfilter Configuration  --->
          Netfilter connection tracking support (M/Y)
	  "connlabel" match support (M)

  or

    CONFIG_NF_CONNTRACK=m or y
    CONFIG_NF_CONNTRACK_LABELS=y
    CONFIG_NETFILTER_XT_MATCH_CONNLABEL=m

Kernel version >= 4.4

  Networking support  --->
    Networking options  --->
    [*] Network packet filtering framework (Netfilter)  --->
        Core Netfilter Configuration  --->
          Netfilter connection tracking support (M/Y)
	  "connlabel" match support (M)
          Number of custom extend (1) (if patched kernel)

  Check your kernel configuration file.
  For non-patched kernel there should be:

    CONFIG_NF_CONNTRACK=m or y
    CONFIG_NF_CONNTRACK_LABELS=y
    CONFIG_NETFILTER_XT_MATCH_CONNLABEL=m
    CONFIG_UNUSED_SYMBOLS=y

  For patched kernel there should be:

    CONFIG_NF_CONNTRACK=m or y
    CONFIG_NF_CONNTRACK_CUSTOM=1
    CONFIG_UNUSED_SYMBOLS=y

----------------------
3. Compile and install
----------------------

The install the modules follow these steps, always from the modules package
root (i.e. where this file is located)

Add option KERNEL_DIR=/path/to/kernel if the kernel sources have non-standard location,
or if you want to compile the module to another kernel configuration.

- To compile the modules first type:

	./autogen.sh
	./configure
	cd  ndpi-netfilter
	make

  For compile with debug messages (not for production)

	./autogen.sh --enable-debug-messages
	./configure
	cd  ndpi-netfilter
	make

- Then, as root, type:

        make modules_install

That would install the nDPI module for the given Linux kernel.
(default /lib/modules/`uname -r`/extra/xt_ndpi.ko)

- Finally, also as root, you should copy the iptables module into the
  corresponding directory. Usually this is "/lib/xtables" or
  "/usr/lib/xtables".

        make install

Now you can read the following section to learn how to use it.

---------------
4. Module usage
---------------

Once you have installed modules ("libxt_ndpi.so" and "xt_ndpi.ko")
you should type (as root):

        modprobe xt_ndpi

If the module has been successfully loaded you shouldn't see any message
(see dmesg for additional info). After loading the kernel module you can
use iptables to add a rule. To see available protocols you can match for
in every packet, type:

        iptables -m ndpi --help

Note that the list is long, you best try: "iptables -m ndpi --help | more"

An example rule would be (as root):

        iptables -A OUTPUT -m ndpi --proto http -j REJECT

This wouldn't allow any HTTP traffic that originates from the machine where the
rule was added.

------------------------------------------------------------------
BIG FAT WARNING!

Do not discard all the unknown traffic. Tcp-syn packets are always treated as an
unknown protocol.
Use the "inprogress" option to determine when the protocol detection process ends.

Example of allowing several protocols:

	iptables -A FORWARD -m ndpi --inprogress ssh,dns,tls -j ACCEPT
	iptables -A FORWARD -m ndpi --clevel dpi --proto ssh,dns,tls -j ACCEPT
	iptables -A FORWARD -m ndpi --unknown -j DROP

Example of disabling several protocols:

	iptables -A FORWARD -m ndpi --inprogress ssh,dns,tls -j ACCEPT
	iptables -A FORWARD -m ndpi --clevel dpi --proto ssh,dns,tls -j DROP
	iptables -A FORWARD -m ndpi --unknown -j ACCEPT

Protocols defined in /proc/net/xt_ndpi/host_proto will be considered at the "dpi" confidence level
Protocols defined in /proc/net/xt_ndpi/ip_proto by ip address have "ip" confidence level.
Protocols defined in /proc/net/xt_ndpi/ip_proto by ip address and port have "user" confidence level.

------------------------------------------------------------------

Using ja4c/tlsfp/tlsv matches.

These options can use only user-defined protocols.
You need to create such a protocol and add hashes with additional prefixes to it.
To check the ja4c hash, add the string JA4C_<hash>. Similar to other options.

Example: 
  echo "add_custom bad_ctcl" >/proc/net/xt_ndpi/proto
  echo "bad_ctls:JA4C_t163c4972d_a11f7a873178_a9ee53acfd9e" >/proc/net/xt_ndpi/host_proto

  iptables -A ... -m ndpi --ja4c bad_ctcl -j DROP


Using NDPI target. To see available options type:

        iptables -j NDPI --help

NDPI target may be using for packet marking (see MARK target), packet classify (see CLASSIFY target)
and selecting traffic for "netflow" (see ipt_netflow).
There is a limitation: the "-j NDPI" target must be after the "-m ndpi" match. 
They can be used in one rule "iptables ... -m ndpi ... -j NDPI ..."
Values for mark or class may be give from /proc/net/xt_ndpi/proto (see procfs section below) or specified in netfilter rules (--values).

Example (traffic shaper)

-----------------------------------------------------------------
#
# HTB setup
#

/etc/htb/eth0
DEFAULT=30
R2Q=160
DCACHE=no

/etc/htb/eth0-2.root
RATE=210Mbit
BURST=20k

/etc/htb/eth0-2:10.high
RATE=100Mbit
CEIL=205Mbit
LEAF=sfq

/etc/htb/eth0-2:20.slow
RATE=20Mbit
CEIL=100Mbit
LEAF=sfq

/etc/htb/eth0-2:30.default
RATE=80Mbit
CEIL=210Mbit
LEAF=sfq

/etc/htb/eth0-2:40.unknown
RATE=1Mbit
CEIL=2Mbit
LEAF=sfq
-----------------------------------------------------------------
modprobe xt_ndpi
# default class 1:30
echo "all 10030/ffffff" >/proc/net/xt_ndpi/proto
for i in ssh pop3 imap smtp pops smtps imaps dns; do
# high speed class 1:10
	echo "$i 10010" >/proc/net/xt_ndpi/proto
done
for i in bittorrent edonkey ftp_data ; do
# low speed class 1:20
	echo "$i 10020" >/proc/net/xt_ndpi/proto
done
echo "unknown 10040" >/proc/net/xt_ndpi/proto

#detect incoming traffic
iptables -t mangle -A INPUT -m ndpi --all
#detect transit traffic
iptables -t mangle -A FORWARD -m ndpi --all
#detect outgoing local traffic
iptables -t mangle -A OUTPUT -m ndpi --all

#traffic classification
iptables -t mangle -A POSTROUTING -o eth0 -j NDPI --ndpi-id --set-clsf

Warning! Do not use the 'mangle/PREROUTING' chain for incoming packets via
the network interface with the translations of addresses (NAT),
since the 'mangle/PREROUTING' chain is performed before NAT. Packet checks need
to be moved to the 'mangle/FORWARD' and 'mangle/INPUT' chains.

----------------
5. Module unload
----------------

In case you want to stop using the nDPI kernel module, first remove every
iptables rule for nDPI and then type (as root):

Without livepatch

        conntrack -F
	rmmod xt_ndpi

With livepatch
        
	conntrack -F
	sleep 1
	echo 0 >/sys/kernel/livepatch/xt_ndpi/enabled
	sleep 1
	rmmod xt_ndpi


--------------------
6. Module parameters
--------------------

xt_debug:	Debug level for xt_ndpi. Default 0. See /proc/net/xt_ndpi/debug
lib_trace:	Maximem debug level for nDPI library. Default 0.
		0-off, 1-error, 2-trace, 3-debug, 4->extra debug"
		(Needed configure with --enable-debug-messages option)

		Use "echo all debug 4 >/proc/net/xt_ndpi/proto"	for 
		maximum debug output.
		This settings may be personality for each protocol.

mtu:    Skip non-linear skb (GRO/LRO) if packet size > mtu. Default 48000.

bt_hash_size:  size of BT hash (2-512) entry*1024. Default 0 (off).
bt6_hash_size: size of BT hash for IPv6 (2-32) entry*1024. Default 0 (off).

bt_hash_timeout: Storage time (sec) of history BT peers. Default 1200.
                 (Only if bt_hash_size > 0).

bt_log_size: size of log BT announces (kb). 192 bytes per announce. Default 128.
                 (Only if bt_hash_size > 0).

ndpi_enable_flow: Enable netflow info. See FLOW_INFO.txt
ndpi_flow_opt:    Output additional info about TLS. See FLOW_INFO.txt
ndpi_flow_limit:  Limit netflow records. Default 10000000 (~4.3Gb RAM). See FLOW_INFO.txt

ndpi_stun_cache: STUN cache control (0-1). Default 0.

tls_buf_size: The maximum buffer size in kB for the TLS protocol. default 8, range 2-16

set_magic_ct: Set new MAGIC_CT value. Default: 0xa55a. Set 0 for random MAGIC_CT
	      If you use conntrackd to synchronize conntrack, you must load the module
	      with the option "set_magic_ct=0" otherwise the kernel will crash.

---------------
7. procfs files
---------------

/proc/net/xt_ndpi/proto - show ID, mark and mask of protocols.

  Use command "cat /proc/net/xt_ndpi/proto" for view settings

  "echo init >/proc/net/xt_ndpi/proto" enable all protocols and set mark by ID and mask 0xff.

  The 'protoid' must be starting from '0x'.
  "bfd" is protocol name.
  "0xbfd" is protocol id.

  Change mark/mask for protocols:
  "echo '(protoid|protoname|all|any) mark[/mask]' >/proc/net/xt_ndpi/proto"

  The prohibition detection protocol:
  "echo '(protoid|protoname) disable' >/proc/net/xt_ndpi/proto"

  !!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!
  !!! We must disable protocols before first using it in iptables  !!!
  !!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!

  Set trace level for protocol:
  "echo '(all|protoid|protoname) debug 0-4' >/proc/net/xt_ndpi/proto"

  Adding user-defined protocol:
  "echo 'add_custom protoname' >/proc/net/xt_ndpi/proto"
  (use ip_proto and host_proto for definition)

  Set name for netnamespace (debug info):
  "echo 'netns xxxxx' >/proc/net/xt_ndpi/proto"

  protoid, mark and mask is hex-value.

  Examples:
  echo all 0/ff >/proc/net/xt_ndpi/proto
  echo http 1 >/proc/net/xt_ndpi/proto
  echo http_proxy 1 >/proc/net/xt_ndpi/proto
  echo 83 1 >/proc/net/xt_ndpi/proto
  echo "add_custom bad_host" >/proc/net/xt_ndpi/proto && \
    echo "bad_host:bad.org" >/proc/net/xt_ndpi/host_proto

/proc/net/xt_ndpi/ip_proto - show and change IP and port based protocols settings

  View current settings:
  "cat /proc/net/xt_ndpi/ip_proto"

  Change settings:
  "echo '[-]prefix ([[(tcp|udp|any):]port[-port]:]protocol[!])+' >/proc/net/xt_ndpi/ip_proto" - change

  Example:      echo '1.1.1.1 any:53:DNS! tcp:81:http' >/proc/net/xt_ndpi/ip_proto

  If checking the first packet did not determine the protocol and there is a "!" after the protocol name,
  then checking the contents of packets is stopped and the specified protocol is assigned to the connection.
  Otherwise, the protocol will be assigned after any of the 3 conditions are met:
  1. 10 or more packets have passed through the connection.
  2. In both directions at least one packet with a payload is transmitted.
  3. Three or more packets with payload are transmitted in one (any) direction.

/proc/net/xt_ndpi/ip6_proto - show and change IPv6 and port based protocols settings. See ip_proto.

/proc/net/xt_ndpi/host_proto - show and change host based settings (used in DNS,HTTP,SSL)

  View current settings:
  "cat /proc/net/xt_ndpi/host_proto"

  Change settings:
  "echo 'protoname:host[,host...][ protoname:host[,host...]]' >/proc/net/xt_ndpi/host_proto"

  If the hostname begins with a "|" character, a match is checked from the beginning of the line.
  If the hostname ends with a "|" character, a match is made to the end of the string.
  "|name|" this is an exact string match.

  Delete match:
  "echo '-protoname:host' >/proc/net/xt_ndpi/host_proto"

  Delete all hosts for the selected protocol:
  "echo '-protoname:*' >/proc/net/xt_ndpi/host_proto"

  Delete all hosts matchs:
  "echo 'reset' >/proc/net/xt_ndpi/host_proto"

  Updating /proc/net/xt_ndpi/host_prot is slow and expensive.
  It is recommended to download all the data in one go.

/proc/net/xt_ndpi/debug - show and change debug output
  
  View current settings:
  "cat /proc/net/xt_ndpi/debug"
  
  Change settings:
  "echo 'packet y' >/proc/net/xt_ndpi/debug"

  Disable all debug:
  "echo 'all y' >/proc/net/xt_ndpi/debug"

/proc/net/xt_ndpi/risks - show and change risk.

  View current settings:

  "cat /proc/net/xt_ndpi/risks"

  Format of /proc/net/xt_ndpi/risks:

  id_number (d|a) description

  Use "echo 'X d' >/proc/net/xt_ndpi/risks" for disable risk with number X.
  Use "echo 'X a' >/proc/net/xt_ndpi/risks" for enable risk with number X.

  Loading custom data for RISKY_DOMAIN, MALICIOUS_JA4, MALICIOUS_SHA1_CERTIFICATE

  We need to create our own protocol with any name (for example "RISKS" ).
  "echo 'add_custom RISKS' >/proc/net/xt_ndpi/proto"

  For RISKY_DOMAIN we need add to host_proto
  "echo 'RISKS:RISK_DOMAIN_bad.domain' >/proc/net/xt_ndpi/host_proto"

  Note :      If the hostname begins with a "|" character, a match is checked against the beginning of the string.
	      If the hostname ends with a "|" character, a match is made to the end of the string.
              "|name|" this is an exact string match.

  For MALICIOUS_JA4 we need add to host_proto
  "echo 'RISKS:RISK_JA4_xxxxxxxxxx_xxxxxxxxxxxx_xxxxxxxxxxxx' >/proc/net/xt_ndpi/host_proto"

  For MALICIOUS_SHA1_CERTIFICATE we need add to host_proto
  "echo 'RISKS:RISK_SHA1CERT_xxxxxxxxxxxxxxxxxxxx' >/proc/net/xt_ndpi/host_proto"

Bittorrents-specific files:

/proc/net/xt_ndpi/announce - show BT announces (from DHT)

/proc/net/xt_ndpi/info[6] - statistics about BT hash. For debug and development.

  "cat /proc/net/xt_ndpi/info" - show number of elements in hash list.
  "echo XXX >/proc/net/xt_ndpi/info && cat /proc/net/xt_ndpi/info" - show ip/port in selected hash list"
  "echo -1 >/proc/net/xt_ndpi/info" - restore view common info

/proc/net/xt_ndpi/cfg - show and change nDPI library parameters.

  "cat /proc/net/xt_ndpi/cfg" - show current settings.
  "echo 'proto_name_or_id param_name value' >/proc/net/xt_ndpi/cfg" - change parameter for specified protocol
  "echo 'param_name value' >/proc/net/xt_ndpi/cfg" - change global parameter

  Example of enabling classification of domain names specified in the certificate [SNI]:

	"echo 'tls subclassification_cert 1' >/proc/net/xt_ndpi/cfg"

/proc/net/xt_ndpi/magic_ct - set or show "magic_ct".

  Changing magic_ct can only be done before loading rules that use ndpi.
  The magic_ct value must be in the range 0-65565.