vrr/ntopng
2
0
Fork 0
mirror of https://github.com/ntop/ntopng.git synced 2026-04-28 15:09:33 +00:00
Code Issues Projects Releases Packages Wiki Activity Actions 3
ntopng/doc/README.clickhouse.md
Alfredo Cardigliano 0c9077ff93
Update README.clickhouse.md
2025-05-05 15:06:08 +02:00

3.6 KiB
Raw Blame History

ClickHouse

For more information about howto use ClickHouse in ntopng please see

Usage Recommendations

We recommend you to read this document, in particular, if your system has limited resources such as memory and disk

Clickhouse Is Eating All My Disk/Memory

You can instruct ntopng to limit disk space usage by setting data retention in preferences to a low value. By default we store 30 days but that can take a lot of disk space on large networks In this case you can reduce it a bit (e.g. to 7 days).

However clickhouse is also using a lot of disk with system tables. You can check how much disk they use with:

SELECT
    table,
    formatReadableSize(sum(bytes)) AS size,
    min(min_date) AS min_date,
    max(max_date) AS max_date
FROM system.parts
WHERE active
GROUP BY table

Query id: 97d9131a-dc97-4b1e-958e-c8e2d00b2c87

┌─table───────────────────┬─size───────┬───min_date─┬───max_date─┐
│ flows                   │ 41.07 GiB  │ 1970-01-01 │ 1970-01-01 │
│ part_log                │ 229.23 MiB │ 2021-11-12 │ 2022-08-09 │
│ metric_log              │ 1.09 GiB   │ 2022-03-15 │ 2022-08-09 │
│ metric_log_3            │ 219.29 MiB │ 2022-02-15 │ 2022-03-15 │
│ trace_log               │ 5.93 GiB   │ 2021-11-12 │ 2022-08-09 │
│ query_thread_log        │ 489.17 MiB │ 2021-11-12 │ 2022-08-09 │
│ system_alerts           │ 18.72 KiB  │ 1970-01-01 │ 1970-01-01 │
│ metric_log_1            │ 112.99 MiB │ 2021-12-16 │ 2021-12-29 │
│ interface_alerts        │ 56.67 KiB  │ 1970-01-01 │ 1970-01-01 │
│ query_log               │ 19.13 MiB  │ 1970-01-01 │ 2022-08-09 │
│ metric_log_2            │ 346.79 MiB │ 2021-12-29 │ 2022-02-15 │
│ host_alerts             │ 10.43 KiB  │ 1970-01-01 │ 1970-01-01 │
│ session_log             │ 71.38 MiB  │ 2021-11-12 │ 2022-08-09 │
│ metric_log_0            │ 237.58 MiB │ 2021-11-12 │ 2021-12-16 │
│ user_alerts             │ 2.02 KiB   │ 1970-01-01 │ 1970-01-01 │
│ asynchronous_metric_log │ 1.25 GiB   │ 1970-01-01 │ 2022-08-09 │
└─────────────────────────┴────────────┴────────────┴────────────┘

You can reduce the table TTL (i.e. how long data is kept in memory) of the system tables, using the following command:

 ALTER TABLE system.XXX MODIFY TTL event_date + INTERVAL 3 DAY;

where XXX is the table you want to use. We suggest to reduce the space used by large tables (column site in the above report), and in particular:

ALTER TABLE system.query_log MODIFY TTL event_date + INTERVAL 3 DAY;
ALTER TABLE system.asynchronous_metric_log MODIFY TTL event_date + INTERVAL 3 DAY;
ALTER TABLE system.metric_log MODIFY TTL event_date + INTERVAL 3 DAY;
ALTER TABLE system.trace_log MODIFY TTL event_date + INTERVAL 3 DAY;

should be enough to reduce disk usage for most setups.

Finally, you need to reduce the disk space used by ClickHouse logs as follows

  • (as root) edit /etc/clickhouse-server/config.xml and under the section do
  • modify trace into error
  • modify 1000M to 100M
  • modify 10 to 3
  • service clickhouse-server start

ClickHouse tips