Documents ClickHouse support

Addresses #6083

doc/src/advanced_features/flows_dump.rst

@@ -24,6 +24,11 @@ When flows dump is enabled, a new `Flow Dump Settings` tab appears in the prefer
 
 These settings are effective for all databases.
 
+ClickHouse
+----------
+
+ntopng integrates with ClickHouse to store historical flows and alerts. ClickHouse is an high-performance SQL database. See :ref:`ClickHouse` for a detailed discussion and guide.
+
 nIndex
 ------
 
@@ -41,6 +46,11 @@ by enabling the *direct* mode. The drawback with this mode is that flows are dum
 before any processing, thus less flow details will be available in the dump as flows are not augmented by ntopng.
 In order to enable this mode `-F nindex;direct` should be specified.
 
+.. warning::
+
+  nIndex support is deprecated and will be discontinued in favor of ClickHouse. ntopng version 5.1 is the last version supporting nIndex.
+  A tool to migrate nIndex to ClickHouse is available at: https://github.com/ntop/ntopng/blob/dev/tools/nindex_export_to_ch.sh
+
 MySQL
 -----
 
@@ -72,13 +82,17 @@ IPv4 and IPv6 flows, respectively.
   A MySQL Table with Dumped Flows
 
 
-.. note::
-
-	MySQL flow explorer is non supported in community edition. We suggest you to use nIndex for high cardinality flow instances.
-
 By enabling MySQL integration, it's also possible to inspect the past flows via
 the ntopng Historical Explorer, which provides many filters and drilldown capabilities.
 
+.. note::
+
+  MySQL flow explorer is non supported in community edition. We suggest you to use nIndex for high cardinality flow instances.
+
+.. warning::
+
+  MySQL flow explorer is deprecated and wil be discontinued in favor of the :ref:`ClickHouse` flows explorer. MySQL dump support will be maintained.
+
 .. figure:: ../img/advanced_features_historical_explorer.png
   :align: center
   :alt: MySQL Historical Explorer

doc/src/cli_options.rst

@@ -132,6 +132,11 @@ ntopng supports a large number of command line parameters. To see what they are,
                                        |   Notes:
                                        |   <facility-text> is case-insensitive.
                                        |
+                                       | clickhouse    Dump in ClickHouse database
+                                       |   Format:
+                                       |   clickhouse;<host[@port]|socket>;<dbname>;<table name>;<user>;<pw>
+                                       |   clickhouse;127.0.0.1;ntopng;flows;default;
+                                       |
                                        | mysql         Dump in MySQL database
                                        |   Format:
                                        |   mysql;<host[@port]|socket>;<dbname>;<table name>;<user>;<pw>

doc/src/clickhouse.rst

@@ -0,0 +1,63 @@
+.. _ClickHouse:
+
+ClickHouse
+##########
+
+ntopng integrates with ClickHouse to store historical flows and alerts. ClickHouse is an high-performance SQL database. To install ClickHouse refer to the official guide_.
+
+.. _guide: https://clickhouse.com/#quick-start
+
+To connect ntopng to ClickHouse use option :code:`-F`. The format of this option is the following
+
+.. code:: bash
+
+    clickhouse;<host[@port]|socket>;<dbname>;<table name>;<user>;<pw>
+
+Where
+
+- :code:`<host[@port]|socket>` Specifies the database :code:`host` or a :code:`socket` file. By default, port :code:`9000` is used for the connection. To use a different port, specify it with :code:`@port`. The host can be a symbolic name or an IP address.
+- :code:`<dbname>` Specifies the name of the database to be used and defaults to :code:`ntopng`
+- :code:`<table name>` Specifies the name of the flows table. Currently, this option is ignored.
+- :code:`<user>` Specifies an user with read and write permissions on :code:`<dbname>`
+- :code:`<pw>` Specifies the password that authenticates :code:`<user>`
+
+Example
+
+To connect ntopng and ClickHouse, both running on the same machine, the following line can be used
+
+.. code:: bash
+
+    ./ntopng -F="clickhouse;127.0.0.1;ntopng;flows;default;default"
+
+In the example above, `127.0.0.1` is used to connect using IPv4 (using the symbolic string :code:`localhost` could resolve to an IPv6 address). A user :code:`default`, identified with password :code:`default`, with read and write permissions on database :code:`ntopng` is indicated as well.
+
+Stored Information
+------------------
+
+ntopng stores historical flows and alerts in ClickHouse.
+
+IPv4 and IPv6 flows are stored in table :code:`flows`. A column :code:`INTERFACE_ID` is used to identify the interface on which the flow was seen, this is useful ntopng is monitoring multiple interfaces (see :code:`-i`).
+
+Alerts are stored in several tables, all ending with suffix :code:`_alerts`. The table prefix indicates the alert family, e.g. :code:`host_alerts` table contains alerts for hosts, :code:`flow_alerts` table contains alerts for flows, and so on.
+
+
+Historical Flows Explorer
+-------------------------
+
+When ClickHouse is enabled, an historical flows explorer becomes available in the ntopng web UI.
+
+.. note::
+
+   ClickHouse support is available on all ntopng versions, however, the historical flows explorer is only available in version Enterprise M or above.
+
+
+The explorer is available from the main menu, under the dashboard section.
+
+.. figure:: ./img/clickhouse_flow_explorer.png
+  :align: center
+  :alt: Historical Flows Explorer
+
+  Historical Flows Explorer
+
+
+

doc/src/faq.rst

@@ -85,7 +85,10 @@ nIndex
 
 The maximum number of interfaces that can be used with nIndex is 16. When more than 16 interfaces are used, those exceeding 16 will work but without nIndex support.
 
+.. warning::
 
+  nIndex support is deprecated and will be discontinued in favor of ClickHouse. ntopng version 5.1 is the last version supporting nIndex.
+  A tool to migrate nIndex to ClickHouse is available at: https://github.com/ntop/ntopng/blob/dev/tools/nindex_export_to_ch.sh
 
 Hardware Sizing
 ===============

doc/src/historical_flows.rst

@@ -3,6 +3,10 @@
 Historical Flow Report
 ======================
 
+.. warning::
+
+  MySQL flow explorer is deprecated and wil be discontinued in favor of the :ref:`ClickHouse` flows explorer. MySQL dump support will be maintained.
+
 ntopng can dump flows data to a persistent storage and provides view to browse
 recorded flows data in the past.
 

doc/src/index.rst

@@ -19,6 +19,7 @@ ntopng can be used in combination with other ntop tools. Documentation to use nt
    self_monitoring/index
    using_with_other_tools/index
    historical_flows
+   clickhouse
    advanced_features/index
    operating_ntopng_on_large_networks
    third_party_integrations/index

doc/src/performances/index.rst

@@ -10,6 +10,3 @@ When ntopng collects flows from nProbe, it is expected to process, without drops
 - 25,5 Kfps per interface with four interfaces (with or without an interface view)
 
 Tests have been executed on an Intel(R) Xeon(R) CPU E3-1230 v5 @ 3.40GHz with 16GB RAM.
-
-When nIndex (:ref:`Flows Dump`) is enabled, expect a decrease of 2-3 Kfps.
-

doc/src/web_gui/settings.rst

@@ -55,7 +55,7 @@ Data retention is expressed in days and it affects:
 
 - Top Talkers stored in sqlite
 - Timeseries
-- Historical Flows dumped with MySQL and nIndex.
+- Historical flows
 
 .. note::