Add HR counters documentation

doc/src/advanced_features/high_resolution_timeseries.rst

@@ -0,0 +1,140 @@
+.. _HighResolutionTimeseries:
+
+High-Resolution Timeseries
+##########################
+
+Overview
+--------
+
+Standard flow records summarise all traffic for the lifetime of a connection into a pair of byte
+and packet counters. This is sufficient for long-term trend analysis, but it hides short-lived
+spikes and makes it impossible to plot throughput at sub-minute granularity.
+
+High-Resolution (HR) timeseries solve this by recording byte counts in **15-second slots** for
+every active flow. Each completed flow stored in ClickHouse therefore carries an embedded timeline
+of its own throughput history, enabling charts at 15-second granularity without a separate
+timeseries database.
+
+.. note::
+
+   High-Resolution timeseries require:
+
+   - An **ntopng Enterprise M** (or better) license.
+   - **ClickHouse** configured as the flow dump backend (``-F clickhouse``). See :ref:`ClickHouse`.
+   - **nProbe** with HR counters support.
+
+Data Flow
+---------
+
+.. code-block:: text
+
+   Physical/virtual interface
+          │
+          ▼
+      nProbe (live capture)
+          │  exports @NTOPNG@ template
+          │  + HR_SRC_TO_DST_BYTES (array of 15s byte buckets)
+          │  + HR_DST_TO_SRC_BYTES (array of 15s byte buckets)
+          │  via ZMQ
+          ▼
+      ntopng (ZMQ collector)
+          │  auto-detects HR fields
+          │  writes consolidated flow record
+          ▼
+      ClickHouse flows table
+          HR_SRC2DST_BYTES  Array(UInt64)
+          HR_DST2SRC_BYTES  Array(UInt64)
+          │
+          ▼
+      Grafana dashboard
+          per-flow bidirectional throughput at 15-second resolution
+
+nProbe Configuration
+--------------------
+
+To enable HR timeseries, add the two HR information elements to the nProbe export template.
+No other nProbe option is required:
+
+.. code-block:: bash
+
+   nprobe -i enp1s0 -n none --zmq "tcp://*:5556" -T "@NTOPNG@ %HR_DST_TO_SRC_BYTES %HR_SRC_TO_DST_BYTES"
+
+ntopng Configuration
+--------------------
+
+No ntopng configuration change is required. ntopng **automatically detects** the presence of HR counters
+in the incoming ZMQ flow records and writes consolidated data to the corresponding ClickHouse columns.
+
+ntopng must be configured to use ClickHouse as the flow dump backend:
+
+.. code-block:: bash
+
+   ntopng -i "tcp://127.0.0.1:5556" -F clickhouse
+
+See :ref:`ClickHouse` for full ClickHouse setup instructions.
+
+ClickHouse Schema
+-----------------
+
+When HR data is present, ntopng stores it in additional columns of the ``flows`` table. Example:
+
+.. list-table::
+   :header-rows: 1
+   :widths: 25 20 55
+
+   * - Column
+     - Type
+     - Description
+   * - ``HR_SRC2DST_BYTES``
+     - ``Array(UInt64)``
+     - 15-second delta byte counters, source-to-destination direction. Element *i* covers the
+       interval ``[FIRST_SEEN + (i·15s), FIRST_SEEN + ((i+1)·15s))``.
+   * - ``HR_DST2SRC_BYTES``
+     - ``Array(UInt64)``
+     - 15-second delta byte counters, destination-to-source direction.
+
+Those columns are empty arrays (``[]``) for flows that were not captured by nProbe with HR
+counters enabled. Existing deployments without HR support are therefore unaffected.
+
+Sample ClickHouse query to read the per-slot throughput of a specific flow:
+
+.. code-block:: sql
+
+   SELECT
+       FLOW_ID,
+       arrayEnumerate(HR_SRC2DST_BYTES) AS slot,
+       arrayElement(HR_SRC2DST_BYTES, slot) AS src2dst_bytes,
+       arrayElement(HR_DST2SRC_BYTES, slot) AS dst2src_bytes
+   FROM flows
+   ARRAY JOIN HR_SRC2DST_BYTES
+   WHERE FLOW_ID = <flow_id>
+   ORDER BY slot;
+
+Grafana Dashboard
+-----------------
+
+A ready-made Grafana dashboard is shipped with ntopng at:
+
+.. code-block:: text
+
+   /usr/share/ntopng/httpdocs/misc/grafana/hr-flow-throughput-dashboard.json
+
+Dashboard title: **ntopng High-Resolution Charts**
+
+The dashboard provides two panels:
+
+- **Service Flow Throughput** — bidirectional throughput for a specific flow, filtered by
+  source IP, destination IP, and destination port. Each data point represents one 15-second slot.
+- **Application Protocol Throughput (All Traffic)** — aggregate throughput broken down by
+  application protocol across all flows in the selected time window.
+
+To import the dashboard into Grafana:
+
+1. In Grafana, go to **Dashboards → Import**.
+2. Upload ``hr-flow-throughput-dashboard.json`` or paste its contents.
+3. When prompted, select the ClickHouse datasource that points to the ntopng database.
+
+The dashboard requires the `Grafana ClickHouse plugin
+<https://grafana.com/grafana/plugins/grafana-clickhouse-datasource/>`_. See
+:ref:`GrafanaIntegration` for general Grafana integration instructions.
+

doc/src/advanced_features/index.rst

@@ -18,4 +18,5 @@ section.
     snapshots
     syslog
     clickhouse_timeseries
+    high_resolution_timeseries
 

doc/src/flow_dump/clickhouse/clickhouse_schema.rst

@@ -181,6 +181,10 @@ Flows table description:
 +---------------------------+---------------------------+----------------+--------------------------------------------------------------------------------------------------------------------------+
 | CLIENT_FINGERPRINT        | Client Fingerprint        | String         | Unique identifier derived from client characteristics                                                                    |
 +---------------------------+---------------------------+----------------+--------------------------------------------------------------------------------------------------------------------------+
+| HR_SRC2DST_BYTES          | HR Src→Dst Bytes          | Array(UInt64)  | 15-second delta byte counters, source-to-destination. Populated when nProbe exports HR_SRC_TO_DST_BYTES. See :ref:`HighResolutionTimeseries`. |
++---------------------------+---------------------------+----------------+--------------------------------------------------------------------------------------------------------------------------+
+| HR_DST2SRC_BYTES          | HR Dst→Src Bytes          | Array(UInt64)  | 15-second delta byte counters, destination-to-source. Populated when nProbe exports HR_DST_TO_SRC_BYTES. See :ref:`HighResolutionTimeseries`. |
++---------------------------+---------------------------+----------------+--------------------------------------------------------------------------------------------------------------------------+
 
 
 The list of the Layer 7 protocols can be found `here <https://github.com/ntop/nDPI/blob/dev/src/include/ndpi_protocol_ids.h>`__

doc/src/using_with_other_tools/nprobe.rst

@@ -72,6 +72,21 @@ Additional fields can be combined with the macro :code:`@NTOPNG@` to specify ext
 
 	  nprobe --zmq "tcp://*:5556" -i eth1 -n none -T "@NTOPNG@ %FLOW_TO_APPLICATION_ID %FLOW_TO_USER_ID %INITIATOR_GW_IP_ADDR %EXPORTER_IPV4_ADDRESS"
 
+High-Resolution Timeseries
+~~~~~~~~~~~~~~~~~~~~~~~~~~
+
+Adding :code:`%HR_SRC_TO_DST_BYTES` and :code:`%HR_DST_TO_SRC_BYTES` to the template enables
+**15-second per-flow byte counters**. ntopng automatically detects these fields and stores them
+in the ClickHouse ``flows`` table for use in high-resolution throughput charts:
+
+.. code:: bash
+
+	  nprobe -i eth1 -n none --zmq "tcp://*:5556" \
+	         -T "@NTOPNG@ %HR_DST_TO_SRC_BYTES %HR_SRC_TO_DST_BYTES"
+
+No ntopng configuration change is required — detection and storage are automatic when ClickHouse
+is configured as the flow dump backend. See :ref:`HighResolutionTimeseries` for full details.
+
 Collecting from Multiple Exporters
 ==================================