mirror of
https://github.com/ntop/ntopng.git
synced 2026-04-28 23:19:33 +00:00
72 lines
3.6 KiB
Markdown
72 lines
3.6 KiB
Markdown
# ClickHouse
|
|
For more information about howto use ClickHouse in ntopng please see
|
|
- https://www.ntop.org/guides/ntopng/flow_dump/clickhouse/clickhouse.html
|
|
|
|
# Usage Recommendations
|
|
We recommend you to [read this document](https://clickhouse.com/docs/en/operations/tips), in particular, if your system has limited resources such as memory and disk
|
|
|
|
|
|
# Clickhouse Is Eating All My Disk/Memory
|
|
You can instruct ntopng to limit disk space usage by setting data retention in preferences to a low value. By default we store 30 days but that can take a lot of disk space on large networks In this case you can reduce it a bit (e.g. to 7 days).
|
|
|
|
However clickhouse is also using a lot of disk with system tables. You can check how much disk they use with:
|
|
|
|
```
|
|
SELECT
|
|
table,
|
|
formatReadableSize(sum(bytes)) AS size,
|
|
min(min_date) AS min_date,
|
|
max(max_date) AS max_date
|
|
FROM system.parts
|
|
WHERE active
|
|
GROUP BY table
|
|
|
|
Query id: 97d9131a-dc97-4b1e-958e-c8e2d00b2c87
|
|
|
|
┌─table───────────────────┬─size───────┬───min_date─┬───max_date─┐
|
|
│ flows │ 41.07 GiB │ 1970-01-01 │ 1970-01-01 │
|
|
│ part_log │ 229.23 MiB │ 2021-11-12 │ 2022-08-09 │
|
|
│ metric_log │ 1.09 GiB │ 2022-03-15 │ 2022-08-09 │
|
|
│ metric_log_3 │ 219.29 MiB │ 2022-02-15 │ 2022-03-15 │
|
|
│ trace_log │ 5.93 GiB │ 2021-11-12 │ 2022-08-09 │
|
|
│ query_thread_log │ 489.17 MiB │ 2021-11-12 │ 2022-08-09 │
|
|
│ system_alerts │ 18.72 KiB │ 1970-01-01 │ 1970-01-01 │
|
|
│ metric_log_1 │ 112.99 MiB │ 2021-12-16 │ 2021-12-29 │
|
|
│ interface_alerts │ 56.67 KiB │ 1970-01-01 │ 1970-01-01 │
|
|
│ query_log │ 19.13 MiB │ 1970-01-01 │ 2022-08-09 │
|
|
│ metric_log_2 │ 346.79 MiB │ 2021-12-29 │ 2022-02-15 │
|
|
│ host_alerts │ 10.43 KiB │ 1970-01-01 │ 1970-01-01 │
|
|
│ session_log │ 71.38 MiB │ 2021-11-12 │ 2022-08-09 │
|
|
│ metric_log_0 │ 237.58 MiB │ 2021-11-12 │ 2021-12-16 │
|
|
│ user_alerts │ 2.02 KiB │ 1970-01-01 │ 1970-01-01 │
|
|
│ asynchronous_metric_log │ 1.25 GiB │ 1970-01-01 │ 2022-08-09 │
|
|
└─────────────────────────┴────────────┴────────────┴────────────┘
|
|
|
|
```
|
|
|
|
You can reduce the table TTL (i.e. how long data is kept in memory) of the system tables, using the following command:
|
|
|
|
```
|
|
ALTER TABLE system.XXX MODIFY TTL event_date + INTERVAL 3 DAY;
|
|
```
|
|
where XXX is the table you want to use. We suggest to reduce the space used by large tables (column site in the above report), and in particular:
|
|
|
|
```
|
|
ALTER TABLE system.query_log MODIFY TTL event_date + INTERVAL 3 DAY;
|
|
ALTER TABLE system.asynchronous_metric_log MODIFY TTL event_date + INTERVAL 3 DAY;
|
|
ALTER TABLE system.metric_log MODIFY TTL event_date + INTERVAL 3 DAY;
|
|
ALTER TABLE system.trace_log MODIFY TTL event_date + INTERVAL 3 DAY;
|
|
```
|
|
|
|
should be enough to reduce disk usage for most setups.
|
|
|
|
Finally, you need to reduce the disk space used by ClickHouse logs as follows
|
|
- (as root) edit /etc/clickhouse-server/config.xml and under the <logger> section do
|
|
- modify <level>trace</level> into <level>error</level>
|
|
- modify <size>1000M</size> to <size>100M</size>
|
|
- modify <count>10</count> to <count>3</count>
|
|
- service clickhouse-server start
|
|
|
|
# ClickHouse tips
|
|
- https://alex.dzyoba.com/kb/clickhouse/
|
|
|