diff --git a/doc/src/advanced_features/flows_dump.rst b/doc/src/advanced_features/flows_dump.rst index 0e076112c3..2c0f440718 100644 --- a/doc/src/advanced_features/flows_dump.rst +++ b/doc/src/advanced_features/flows_dump.rst @@ -24,6 +24,11 @@ When flows dump is enabled, a new `Flow Dump Settings` tab appears in the prefer These settings are effective for all databases. +ClickHouse +---------- + +ntopng integrates with ClickHouse to store historical flows and alerts. ClickHouse is an high-performance SQL database. See :ref:`ClickHouse` for a detailed discussion and guide. + nIndex ------ @@ -41,6 +46,11 @@ by enabling the *direct* mode. The drawback with this mode is that flows are dum before any processing, thus less flow details will be available in the dump as flows are not augmented by ntopng. In order to enable this mode `-F nindex;direct` should be specified. +.. warning:: + + nIndex support is deprecated and will be discontinued in favor of ClickHouse. ntopng version 5.1 is the last version supporting nIndex. + A tool to migrate nIndex to ClickHouse is available at: https://github.com/ntop/ntopng/blob/dev/tools/nindex_export_to_ch.sh + MySQL ----- @@ -72,13 +82,17 @@ IPv4 and IPv6 flows, respectively. A MySQL Table with Dumped Flows -.. note:: - - MySQL flow explorer is non supported in community edition. We suggest you to use nIndex for high cardinality flow instances. - By enabling MySQL integration, it's also possible to inspect the past flows via the ntopng Historical Explorer, which provides many filters and drilldown capabilities. +.. note:: + + MySQL flow explorer is non supported in community edition. We suggest you to use nIndex for high cardinality flow instances. + +.. warning:: + + MySQL flow explorer is deprecated and wil be discontinued in favor of the :ref:`ClickHouse` flows explorer. MySQL dump support will be maintained. + .. figure:: ../img/advanced_features_historical_explorer.png :align: center :alt: MySQL Historical Explorer diff --git a/doc/src/cli_options.rst b/doc/src/cli_options.rst index 56c58966c1..930a4d2c33 100644 --- a/doc/src/cli_options.rst +++ b/doc/src/cli_options.rst @@ -132,6 +132,11 @@ ntopng supports a large number of command line parameters. To see what they are, | Notes: | is case-insensitive. | + | clickhouse Dump in ClickHouse database + | Format: + | clickhouse;;;;; + | clickhouse;127.0.0.1;ntopng;flows;default; + | | mysql Dump in MySQL database | Format: | mysql;;;
;; diff --git a/doc/src/clickhouse.rst b/doc/src/clickhouse.rst new file mode 100644 index 0000000000..e118a1698f --- /dev/null +++ b/doc/src/clickhouse.rst @@ -0,0 +1,63 @@ +.. _ClickHouse: + +ClickHouse +########## + +ntopng integrates with ClickHouse to store historical flows and alerts. ClickHouse is an high-performance SQL database. To install ClickHouse refer to the official guide_. + +.. _guide: https://clickhouse.com/#quick-start + +To connect ntopng to ClickHouse use option :code:`-F`. The format of this option is the following + +.. code:: bash + + clickhouse;;;
;; + +Where + +- :code:`` Specifies the database :code:`host` or a :code:`socket` file. By default, port :code:`9000` is used for the connection. To use a different port, specify it with :code:`@port`. The host can be a symbolic name or an IP address. +- :code:`` Specifies the name of the database to be used and defaults to :code:`ntopng` +- :code:`
` Specifies the name of the flows table. Currently, this option is ignored. +- :code:`` Specifies an user with read and write permissions on :code:`` +- :code:`` Specifies the password that authenticates :code:`` + +Example + +To connect ntopng and ClickHouse, both running on the same machine, the following line can be used + +.. code:: bash + + ./ntopng -F="clickhouse;127.0.0.1;ntopng;flows;default;default" + +In the example above, `127.0.0.1` is used to connect using IPv4 (using the symbolic string :code:`localhost` could resolve to an IPv6 address). A user :code:`default`, identified with password :code:`default`, with read and write permissions on database :code:`ntopng` is indicated as well. + +Stored Information +------------------ + +ntopng stores historical flows and alerts in ClickHouse. + +IPv4 and IPv6 flows are stored in table :code:`flows`. A column :code:`INTERFACE_ID` is used to identify the interface on which the flow was seen, this is useful ntopng is monitoring multiple interfaces (see :code:`-i`). + +Alerts are stored in several tables, all ending with suffix :code:`_alerts`. The table prefix indicates the alert family, e.g. :code:`host_alerts` table contains alerts for hosts, :code:`flow_alerts` table contains alerts for flows, and so on. + + +Historical Flows Explorer +------------------------- + +When ClickHouse is enabled, an historical flows explorer becomes available in the ntopng web UI. + +.. note:: + + ClickHouse support is available on all ntopng versions, however, the historical flows explorer is only available in version Enterprise M or above. + + +The explorer is available from the main menu, under the dashboard section. + +.. figure:: ./img/clickhouse_flow_explorer.png + :align: center + :alt: Historical Flows Explorer + + Historical Flows Explorer + + + diff --git a/doc/src/faq.rst b/doc/src/faq.rst index 77611ec099..531a3c84de 100644 --- a/doc/src/faq.rst +++ b/doc/src/faq.rst @@ -85,7 +85,10 @@ nIndex The maximum number of interfaces that can be used with nIndex is 16. When more than 16 interfaces are used, those exceeding 16 will work but without nIndex support. +.. warning:: + nIndex support is deprecated and will be discontinued in favor of ClickHouse. ntopng version 5.1 is the last version supporting nIndex. + A tool to migrate nIndex to ClickHouse is available at: https://github.com/ntop/ntopng/blob/dev/tools/nindex_export_to_ch.sh Hardware Sizing =============== diff --git a/doc/src/historical_flows.rst b/doc/src/historical_flows.rst index 1c459a639f..bbc8388e8a 100644 --- a/doc/src/historical_flows.rst +++ b/doc/src/historical_flows.rst @@ -3,6 +3,10 @@ Historical Flow Report ====================== +.. warning:: + + MySQL flow explorer is deprecated and wil be discontinued in favor of the :ref:`ClickHouse` flows explorer. MySQL dump support will be maintained. + ntopng can dump flows data to a persistent storage and provides view to browse recorded flows data in the past. diff --git a/doc/src/img/clickhouse_flow_explorer.png b/doc/src/img/clickhouse_flow_explorer.png new file mode 100644 index 0000000000..90f86f4ce3 Binary files /dev/null and b/doc/src/img/clickhouse_flow_explorer.png differ diff --git a/doc/src/index.rst b/doc/src/index.rst index e04b49b94e..48e35e5bce 100644 --- a/doc/src/index.rst +++ b/doc/src/index.rst @@ -19,6 +19,7 @@ ntopng can be used in combination with other ntop tools. Documentation to use nt self_monitoring/index using_with_other_tools/index historical_flows + clickhouse advanced_features/index operating_ntopng_on_large_networks third_party_integrations/index diff --git a/doc/src/performances/index.rst b/doc/src/performances/index.rst index 155c4a03db..f220ada199 100644 --- a/doc/src/performances/index.rst +++ b/doc/src/performances/index.rst @@ -10,6 +10,3 @@ When ntopng collects flows from nProbe, it is expected to process, without drops - 25,5 Kfps per interface with four interfaces (with or without an interface view) Tests have been executed on an Intel(R) Xeon(R) CPU E3-1230 v5 @ 3.40GHz with 16GB RAM. - -When nIndex (:ref:`Flows Dump`) is enabled, expect a decrease of 2-3 Kfps. - diff --git a/doc/src/web_gui/settings.rst b/doc/src/web_gui/settings.rst index dcfcb891bb..8b3f56c93e 100644 --- a/doc/src/web_gui/settings.rst +++ b/doc/src/web_gui/settings.rst @@ -55,7 +55,7 @@ Data retention is expressed in days and it affects: - Top Talkers stored in sqlite - Timeseries -- Historical Flows dumped with MySQL and nIndex. +- Historical flows .. note::