diff --git a/doc/README.clickhouse_schema.md b/doc/README.clickhouse_schema.md new file mode 100644 index 0000000000..4acd472827 --- /dev/null +++ b/doc/README.clickhouse_schema.md @@ -0,0 +1,279 @@ +# ntopng ClickHouse Database Schema + +This document describes the ClickHouse database schema used by ntopng to persist flow records, alerts, assets, and related analytical data. The schema is defined in two files that must be kept in sync: + +| File | Purpose | +|------|---------| +| `httpdocs/misc/db_schema_clickhouse.sql` | Single-node ClickHouse deployment (uses `MergeTree` family engines) | +| `httpdocs/misc/db_schema_clickhouse_cluster.sql` | Multi-node replicated cluster deployment (uses `ReplicatedMergeTree` / `ReplicatedReplacingMergeTree` engines and `ON CLUSTER '$CLUSTER'` DDL) | + +Both files are parsed and executed by ntopng at startup via `pro/src/ClickHouseDB.cpp`. Statements are separated by `@` delimiters rather than standard semicolons to allow ntopng's SQL runner to feed them one at a time. Each file contains both `CREATE TABLE IF NOT EXISTS` statements (idempotent schema bootstrap) and `ALTER TABLE … ADD/MODIFY COLUMN IF NOT EXISTS` statements (idempotent schema migrations for upgrades from older versions). + +--- + +## Table Overview + +| Table | Engine | Description | +|-------|--------|-------------| +| [`flows`](#flows) | MergeTree | Per-flow telemetry records | +| [`hourly_flows`](#hourly_flows) | MergeTree | Hourly aggregated flow summaries | +| [`hourly_asn`](#hourly_asn) | MergeTree | Hourly per-ASN traffic statistics | +| [`active_monitoring_alerts`](#alert-tables) | MergeTree | Historical active-monitoring alerts | +| [`host_alerts`](#alert-tables) | MergeTree | Historical host alerts | +| [`mac_alerts`](#alert-tables) | MergeTree | Historical MAC/device alerts | +| [`snmp_alerts`](#alert-tables) | MergeTree | Historical SNMP alerts | +| [`network_alerts`](#alert-tables) | MergeTree | Historical subnet alerts | +| [`as_alerts`](#alert-tables) | MergeTree | Historical Autonomous System alerts | +| [`interface_alerts`](#alert-tables) | MergeTree | Historical interface alerts | +| [`user_alerts`](#alert-tables) | MergeTree | Historical user alerts | +| [`system_alerts`](#alert-tables) | MergeTree | Historical system alerts | +| [`engaged_*_alerts`](#engaged-alert-tables) | Memory | Currently active (engaged) alerts | +| [`vulnerability_scan_data`](#vulnerability_scan_data) | MergeTree | Per-host vulnerability scan results | +| [`vulnerability_scan_report`](#vulnerability_scan_report) | MergeTree | Vulnerability scan summary reports | +| [`mitre_table_info`](#mitre_table_info) | ReplacingMergeTree | MITRE ATT&CK mappings | +| [`assets`](#assets) | ReplacingMergeTree | Network asset inventory | + +Views that merge historical and engaged tables are described in the [Views](#views) section. + +--- + +## flows + +The central table. Every bidirectional network flow observed by ntopng — whether captured locally or received via NetFlow/sFlow/IPFIX — produces one row. Partitioned by day on `FIRST_SEEN`; ordered by `(FIRST_SEEN, IPV4_SRC_ADDR, IPV4_DST_ADDR)`. + +### Key column groups + +| Group | Columns | +|-------|---------| +| Identity | `FLOW_ID`, `NTOPNG_INSTANCE_NAME`, `INTERFACE_ID` | +| Timing | `FIRST_SEEN`, `LAST_SEEN` | +| Network 5-tuple | `PROTOCOL`, `IPV4_SRC_ADDR` / `IPV6_SRC_ADDR`, `IP_SRC_PORT`, `IPV4_DST_ADDR` / `IPV6_DST_ADDR`, `IP_DST_PORT` | +| Traffic counters | `PACKETS`, `TOTAL_BYTES`, `SRC2DST_BYTES`, `DST2SRC_BYTES`, `SRC2DST_PACKETS`, `DST2SRC_PACKETS` | +| Layer 7 | `L7_PROTO`, `L7_PROTO_MASTER`, `L7_CATEGORY`, `INFO`, `FLOW_RISK` | +| QoS / latency | `SRC2DST_DSCP`, `DST2SRC_DSCP`, `CLIENT_NW_LATENCY_US`, `SERVER_NW_LATENCY_US`, `QOE_SCORE` | +| Host metadata | `SRC_LABEL`, `DST_LABEL`, `SRC_COUNTRY_CODE`, `DST_COUNTRY_CODE`, `SRC_ASN`, `DST_ASN`, `SRC_PEER_ASN`, `DST_PEER_ASN`, `SRC_MAC`, `DST_MAC` | +| Network topology | `VLAN_ID`, `COMMUNITY_ID`, `OBSERVATION_POINT_ID`, `INTERFACE_ROLE`, `PROBE_IP`, `EXPORTER_SITE`, `INPUT_SNMP`, `OUTPUT_SNMP`, `SRC_NETWORK_ID`, `DST_NETWORK_ID` | +| Host pools | `SRC_HOST_POOL_ID`, `DST_HOST_POOL_ID` | +| Process info (eBPF) | `SRC_PROC_NAME`, `DST_PROC_NAME`, `SRC_PROC_USER_NAME`, `DST_PROC_USER_NAME` | +| TCP state | `SRC2DST_TCP_FLAGS`, `DST2SRC_TCP_FLAGS`, `MINOR_CONNECTION_STATE`, `MAJOR_CONNECTION_STATE` | +| Fingerprints | `CLIENT_FINGERPRINT`, `TCP_FINGERPRINT` | +| NAT | `POST_NAT_IPV4_SRC_ADDR`, `POST_NAT_SRC_PORT`, `POST_NAT_IPV4_DST_ADDR`, `POST_NAT_DST_PORT` | +| Wireless | `WLAN_SSID`, `WTP_MAC_ADDRESS` | +| Alert fields | `STATUS`, `SCORE`, `SEVERITY`, `ALERT_STATUS`, `ALERT_CATEGORY`, `ALERT_JSON`, `ALERTS_MAP`, `IS_ALERT_DELETED`, `FLOW_RISK` | +| Threat intel flags | `IS_CLI_ATTACKER`, `IS_CLI_VICTIM`, `IS_CLI_BLACKLISTED`, `IS_SRV_ATTACKER`, `IS_SRV_VICTIM`, `IS_SRV_BLACKLISTED` | +| User annotation | `USER_LABEL`, `USER_LABEL_TSTAMP`, `PROFILE`, `REQUIRE_ATTENTION` | +| Protocol detail | `PROTOCOL_INFO_JSON`, `DOMAIN_NAME` | + +> **Note on alert flows**: A flow row doubles as an alert record when `STATUS != 0`. The `flow_alerts_view` filters for these rows and joins them with `mitre_table_info` for ATT&CK enrichment. + +> **Country encoding**: `SRC_COUNTRY_CODE` and `DST_COUNTRY_CODE` store two ISO 3166-1 ASCII letters packed into a `UInt16` (high byte = first letter). The `flow_alerts_view` decodes them with `char(bitShiftRight(..., 8), bitAnd(..., 0xFF))`. + +--- + +## hourly_flows + +Hourly rollup of the `flows` table. Multiple raw flows sharing the same 5-tuple within an hour are collapsed into one row with summed byte/packet counters. Used for long-term trend analysis and dashboards that do not need per-flow granularity. + +Compared to `flows` it omits per-flow alert fields, process info, fingerprints, TCP flags, and NAT columns, and adds: + +- `NUM_FLOWS UInt32` — number of raw flows collapsed into this row. + +--- + +## hourly_asn + +Hourly aggregated traffic statistics broken down by source/destination ASN pair. Used for AS-level traffic analysis and BGP peer analytics. Partitioned by day on `FIRST_SEEN`; ordered by `(FIRST_SEEN, SRC_ASN, DST_ASN)`. + +Columns include `SRC_ASN`, `DST_ASN`, `SRC_PEER_ASN`, `DST_PEER_ASN`, directional byte/packet counters, `PROBE_IP`, and SNMP interface indices. + +--- + +## Alert Tables + +ntopng models nine entity types that can generate alerts. Each entity type has a pair of tables: + +| Historical table | Engaged (in-memory) table | View | +|-----------------|--------------------------|------| +| `active_monitoring_alerts` | `engaged_active_monitoring_alerts` | `active_monitoring_alerts_view` | +| `host_alerts` | `engaged_host_alerts` | `host_alerts_view` | +| `mac_alerts` | `engaged_mac_alerts` | `mac_alerts_view` | +| `snmp_alerts` | `engaged_snmp_alerts` | `snmp_alerts_view` | +| `network_alerts` | `engaged_network_alerts` | `network_alerts_view` | +| `as_alerts` | `engaged_as_alerts` | `as_alerts_view` | +| `interface_alerts` | `engaged_interface_alerts` | `interface_alerts_view` | +| `user_alerts` | `engaged_user_alerts` | `user_alerts_view` | +| `system_alerts` | `engaged_system_alerts` | `system_alerts_view` | + +### Common columns (all alert tables) + +| Column | Type | Description | +|--------|------|-------------| +| `rowid` | UUID | Unique row identifier | +| `alert_id` | UInt32 | Alert type (maps to ntopng alert type enum) | +| `alert_status` | UInt8 | Lifecycle state: 0 = engaged/active, 1 = released/archived | +| `interface_id` | UInt16 | ntopng interface; 65535 = system/global scope | +| `tstamp` | DateTime | Alert start time | +| `tstamp_end` | DateTime | Alert resolution time (epoch zero if still active) | +| `severity` | UInt8 | Severity level (maps to `AlertLevel` enum) | +| `score` | UInt16 | Numeric risk/impact score | +| `granularity` | UInt8 | Check interval that triggered the alert (1=1min, 2=5min, …) | +| `counter` | UInt32 | Consecutive detection count | +| `description` | String | Human-readable description | +| `json` | String | Additional context as JSON | +| `user_label` | String | User-defined free-text label | +| `user_label_tstamp` | DateTime | When `user_label` was last set | +| `alert_category` | UInt8 | Category (maps to `AlertCategory` enum) | +| `require_attention` | Boolean | Flagged for manual review | + +### Entity-specific columns + +**`active_monitoring_alerts`**: `resolved_ip`, `resolved_name`, `measurement`, `measure_threshold`, `measure_value` + +**`host_alerts`**: `ip_version`, `ip`, `vlan_id`, `name`, `is_attacker`, `is_victim`, `is_client`, `is_server`, `host_pool_id`, `network`, `country` + +**`mac_alerts`**: `address`, `device_type`, `name`, `is_attacker`, `is_victim` + +**`snmp_alerts`**: `ip`, `port` (ifIndex), `name`, `port_name` + +**`network_alerts`**: `local_network_id`, `name`, `alias` + +**`as_alerts`**: `asn`, `name`, `alias` + +**`interface_alerts`**: `ifid`, `subtype`, `name`, `alias` + +**`user_alerts`**: `user` + +**`system_alerts`**: `name` + +--- + +## Engaged Alert Tables + +Each `engaged_*_alerts` table uses the `Memory` engine and holds only the currently-firing (engaged) alerts. Rows are inserted when an alert fires and deleted when it resolves. They have identical column schemas to their MergeTree counterparts. + +These tables are dropped and recreated on every ntopng startup to prevent stale engaged alerts from persisting across restarts. + +--- + +## vulnerability_scan_data + +Stores raw per-host vulnerability scan results produced by the ntopng Vulnerability Scanner (VS) module. Partitioned by day on `LAST_SCAN`; ordered by `(LAST_SCAN, HOST, SCAN_TYPE)`. + +| Column | Description | +|--------|-------------| +| `HOST` | IP address or hostname of the scanned target | +| `SCAN_TYPE` | Scan type (e.g. `nmap`, `openvas`) | +| `LAST_SCAN` | Timestamp of the most recent scan | +| `JSON_INFO` | Full scan results as JSON | +| `VS_RESULT_FILE` | Path to the raw result file on disk | + +--- + +## vulnerability_scan_report + +One row per completed vulnerability scan report. Partitioned by day on `REPORT_DATE`. + +| Column | Description | +|--------|-------------| +| `REPORT_NAME` | User-defined report name | +| `REPORT_DATE` | Report generation timestamp | +| `REPORT_JSON_INFO` | Full report metadata as JSON | +| `NUM_SCANNED_HOSTS` | Hosts scanned | +| `NUM_CVES` | CVEs identified | +| `NUM_TCP_PORTS` | Open TCP ports found | +| `NUM_UDP_PORTS` | Open UDP ports found | + +--- + +## mitre_table_info + +Maps ntopng alert type IDs and entity type IDs to MITRE ATT&CK framework entries. Uses `ReplacingMergeTree` keyed on `(ALERT_ID, ENTITY_ID)` to prevent duplicates. + +| Column | Description | +|--------|-------------| +| `ALERT_ID` | ntopng alert type ID | +| `ENTITY_ID` | ntopng entity type ID (1=host, 4=flow, …) | +| `TACTIC` | MITRE ATT&CK tactic identifier | +| `TECHNIQUE` | MITRE ATT&CK technique identifier | +| `SUB_TECHNIQUE` | Sub-technique identifier (0 if none) | +| `MITRE_ID` | ATT&CK ID string (e.g. `T1046`, `T1595.002`) | + +This table is joined by `host_alerts_view` and `flow_alerts_view` to surface MITRE context alongside alert data. + +--- + +## assets + +Network asset inventory. One row per discovered or imported asset (host, MAC address, network device). Uses `ReplacingMergeTree(version)` so that re-observations update existing rows rather than creating duplicates. Primary key is `(type, key)`. + +| Column | Description | +|--------|-------------| +| `type` | Asset category (e.g. `host`, `mac`, `network_device`) | +| `key` | Unique key within the type (IP address, MAC address, etc.) | +| `ifid` | ntopng interface where the asset was observed | +| `ip` | IP address (empty if not applicable) | +| `mac` | MAC address | +| `vlan` | VLAN (0 if untagged) | +| `network` | ntopng local-network ID | +| `name` | Resolved hostname or user-defined name | +| `device_type` | Device category (maps to `DeviceType` enum) | +| `manufacturer` | Manufacturer from MAC OUI lookup | +| `first_seen` | First observation timestamp | +| `last_seen` | Most recent observation timestamp | +| `gateway_mac` | MAC address of the gateway used to reach this asset | +| `json_info` | Additional metadata as JSON (OS info, open ports, etc.) | +| `version` | Monotonic counter used by `ReplacingMergeTree` for deduplication | +| `os_type` | Detected operating system type | +| `model` | Hardware model string | + +--- + +## Views + +Views are recreated on each ntopng startup (`DROP VIEW IF EXISTS` + `CREATE VIEW IF NOT EXISTS`). + +### Per-entity alert views + +Each `*_alerts_view` is a `UNION ALL` of its historical MergeTree table and its in-memory engaged counterpart, providing a unified query surface for both resolved and currently-active alerts: + +```sql +-- example +SELECT * FROM host_alerts +UNION ALL +SELECT * FROM engaged_host_alerts +``` + +**`host_alerts_view`** additionally `LEFT JOIN`s `mitre_table_info` (on `ENTITY_ID = 1`) to attach ATT&CK tactic/technique columns. + +### flow_alerts_view + +Selects only alert flows from the `flows` table (`STATUS != 0 AND IS_ALERT_DELETED != 1`), renames columns to a friendlier lowercase schema (e.g. `IPV4_SRC_ADDR` → resolved `cli_ip` string), and `LEFT JOIN`s `mitre_table_info` (on `ENTITY_ID = 4`). + +### all_alerts_view + +A single `UNION ALL` across all per-entity alert views plus alert flows, exposing a minimal common schema: `entity_id`, `interface_id`, `alert_id`, `alert_status`, `require_attention`, `tstamp`, `tstamp_end`, `severity`, `score`, `alert_category`. Entity ID values: + +| `entity_id` | Source | +|-------------|--------| +| 0 | `interface_alerts_view` | +| 1 | `host_alerts_view` | +| 2 | `network_alerts_view` | +| 3 | `snmp_alerts_view` | +| 4 | `flows` (alert rows) | +| 5 | `mac_alerts_view` | +| 7 | `user_alerts_view` | +| 8 | `active_monitoring_alerts_view` | +| 9 | `system_alerts_view` | +| 10 | `as_alerts_view` | + +--- + +## Schema Versioning and Migrations + +The schema files follow an additive migration pattern: new columns are added with `ALTER TABLE … ADD COLUMN IF NOT EXISTS` statements appended at the end of the file. Columns are never removed via migration (only via explicit `DROP COLUMN IF EXISTS` for columns that were renamed or replaced). This ensures forward compatibility — an older ntopng version can read a database created by a newer version without crashing, it will simply ignore unknown columns. + +When adding a new column to the schema: +1. Add it to the `CREATE TABLE` statement. +2. Add a corresponding `ALTER TABLE … ADD COLUMN IF NOT EXISTS` migration statement. +3. Keep `db_schema_clickhouse.sql` and `db_schema_clickhouse_cluster.sql` in sync. +4. Also update `pro/include/FlowsTable.h`, `pro/src/ClickHouseDB.cpp`, `scripts/lua/modules/tag_utils.lua`, and `scripts/lua/modules/historical_flow_utils.lua` as noted in the header comment of the schema files. diff --git a/httpdocs/misc/db_schema_clickhouse.sql b/httpdocs/misc/db_schema_clickhouse.sql index 241dd01525..2021d45e2b 100644 --- a/httpdocs/misc/db_schema_clickhouse.sql +++ b/httpdocs/misc/db_schema_clickhouse.sql @@ -4,98 +4,100 @@ /* - pro/src/ClickHouseDB.cpp -> ClickHouseDB::dumpFlow() */ /* - scripts/lua/modules/tag_utils.lua -> By adding the new tags for filters */ /* - scripts/lua/modules/historical_flow_utils.lua -> Columns mapping */ +/* - doc/README.clickhouse_schema.md */ CREATE TABLE IF NOT EXISTS `flows` ( -`FLOW_ID` UInt64, -`IP_PROTOCOL_VERSION` UInt8, -`FIRST_SEEN` DateTime, -`LAST_SEEN` DateTime, -`VLAN_ID` UInt16, /* LowCardinality */ -`PACKETS` UInt32, -`TOTAL_BYTES` UInt64, -`SRC2DST_BYTES` UInt64, -`DST2SRC_BYTES` UInt64, -`SRC2DST_DSCP` UInt8, -`DST2SRC_DSCP` UInt8, -`PROTOCOL` UInt8, -`IPV4_SRC_ADDR` UInt32, -`IPV6_SRC_ADDR` IPv6, -`IP_SRC_PORT` UInt16, -`IPV4_DST_ADDR` UInt32, -`IPV6_DST_ADDR` IPv6, -`IP_DST_PORT` UInt16, -`L7_PROTO` UInt16, -`L7_PROTO_MASTER` UInt16, -`L7_CATEGORY` UInt16, -`FLOW_RISK` UInt64, -`INFO` String, -`PROFILE` String, -`NTOPNG_INSTANCE_NAME` String, -`INTERFACE_ID` UInt16, -`STATUS` UInt8, -`SRC_COUNTRY_CODE` UInt16, -`DST_COUNTRY_CODE` UInt16, -`SRC_LABEL` String, -`DST_LABEL` String, -`SRC_MAC` UInt64, -`DST_MAC` UInt64, -`COMMUNITY_ID` String, -`SRC_ASN` UInt32, -`DST_ASN` UInt32, -`PROBE_IP` UInt32, /* EXPORTER_IPV4_ADDRESS */ -`EXPORTER_SITE` UInt16, -`INTERFACE_ROLE` UInt8, -`OBSERVATION_POINT_ID` UInt16, -`SRC2DST_TCP_FLAGS` UInt8, -`DST2SRC_TCP_FLAGS` UInt8, -`SCORE` UInt16, -`QOE_SCORE` UInt8, -`CLIENT_NW_LATENCY_US` UInt32, -`SERVER_NW_LATENCY_US` UInt32, -`CLIENT_LOCATION` UInt8, -`SERVER_LOCATION` UInt8, -`SRC_NETWORK_ID` UInt32, -`DST_NETWORK_ID` UInt32, -`CLIENT_FINGERPRINT` String, -`TCP_FINGERPRINT` String, -`INPUT_SNMP` UInt32, -`OUTPUT_SNMP` UInt32, -`SRC_HOST_POOL_ID` UInt16, -`DST_HOST_POOL_ID` UInt16, -`SRC_PROC_NAME` String, -`DST_PROC_NAME` String, -`SRC_PROC_USER_NAME` String, -`DST_PROC_USER_NAME` String, -`ALERTS_MAP` String, -`SEVERITY` UInt8, -`IS_CLI_ATTACKER` UInt8, -`IS_CLI_VICTIM` UInt8, -`IS_CLI_BLACKLISTED` UInt8, -`IS_SRV_ATTACKER` UInt8, -`IS_SRV_VICTIM` UInt8, -`IS_SRV_BLACKLISTED` UInt8, -`ALERT_STATUS` UInt8, -`USER_LABEL` String, -`USER_LABEL_TSTAMP` DateTime, -`PROTOCOL_INFO_JSON` String, -`ALERT_JSON` String, -`IS_ALERT_DELETED` UInt8, -`SRC2DST_PACKETS` UInt32, -`DST2SRC_PACKETS` UInt32, -`ALERT_CATEGORY` UInt8, -`MINOR_CONNECTION_STATE` UInt8, -`MAJOR_CONNECTION_STATE` UInt8, -`POST_NAT_IPV4_SRC_ADDR` UInt32, -`POST_NAT_SRC_PORT` UInt32, -`POST_NAT_IPV4_DST_ADDR` UInt32, -`POST_NAT_DST_PORT` UInt32, -`WLAN_SSID` String, -`WTP_MAC_ADDRESS` UInt64, -`DOMAIN_NAME` String, -`SRC_PEER_ASN` UInt32, -`DST_PEER_ASN` UInt32, -`REQUIRE_ATTENTION` Boolean -) ENGINE = MergeTree() PARTITION BY toYYYYMMDD(FIRST_SEEN) ORDER BY (FIRST_SEEN, IPV4_SRC_ADDR, IPV4_DST_ADDR); +`FLOW_ID` UInt64 COMMENT 'Unique flow identifier assigned by ntopng', +`IP_PROTOCOL_VERSION` UInt8 COMMENT 'IP version: 4 for IPv4, 6 for IPv6', +`FIRST_SEEN` DateTime COMMENT 'Timestamp of the first packet of the flow', +`LAST_SEEN` DateTime COMMENT 'Timestamp of the last packet of the flow', +`VLAN_ID` UInt16 /* LowCardinality */ COMMENT '802.1Q VLAN tag (0 if untagged)', +`PACKETS` UInt32 COMMENT 'Total packet count in both directions', +`TOTAL_BYTES` UInt64 COMMENT 'Total bytes transferred in both directions', +`SRC2DST_BYTES` UInt64 COMMENT 'Bytes sent from client (source) to server (destination)', +`DST2SRC_BYTES` UInt64 COMMENT 'Bytes sent from server (destination) to client (source)', +`SRC2DST_DSCP` UInt8 COMMENT 'DSCP value observed in the client-to-server direction', +`DST2SRC_DSCP` UInt8 COMMENT 'DSCP value observed in the server-to-client direction', +`PROTOCOL` UInt8 COMMENT 'IP transport protocol number (6=TCP, 17=UDP, 1=ICMP, etc.)', +`IPV4_SRC_ADDR` UInt32 COMMENT 'Source IPv4 address as a 32-bit integer; 0 for IPv6 flows', +`IPV6_SRC_ADDR` IPv6 COMMENT 'Source IPv6 address; all-zeros for IPv4 flows', +`IP_SRC_PORT` UInt16 COMMENT 'Source (client) port number', +`IPV4_DST_ADDR` UInt32 COMMENT 'Destination IPv4 address as a 32-bit integer; 0 for IPv6 flows', +`IPV6_DST_ADDR` IPv6 COMMENT 'Destination IPv6 address; all-zeros for IPv4 flows', +`IP_DST_PORT` UInt16 COMMENT 'Destination (server) port number', +`L7_PROTO` UInt16 COMMENT 'nDPI layer-7 application protocol identifier', +`L7_PROTO_MASTER` UInt16 COMMENT 'nDPI master/carrier protocol ID (e.g. TLS when L7_PROTO is HTTPS)', +`L7_CATEGORY` UInt16 COMMENT 'nDPI application category identifier', +`FLOW_RISK` UInt64 COMMENT 'Bitmap of nDPI flow risk flags (each bit represents a distinct risk)', +`INFO` String COMMENT 'Supplementary flow info extracted by nDPI (e.g. HTTP host, DNS query name, TLS SNI)', +`PROFILE` String COMMENT 'Traffic policy profile name matched by this flow', +`NTOPNG_INSTANCE_NAME` String COMMENT 'Hostname/name of the ntopng instance that captured this flow', +`INTERFACE_ID` UInt16 COMMENT 'ntopng internal interface identifier', +`STATUS` UInt8 COMMENT 'Flow alert type ID (0 = normal non-alert flow; non-zero = alert type)', +`SRC_COUNTRY_CODE` UInt16 COMMENT 'Source IP geo-country: two ASCII letters packed into a UInt16 (high byte = first letter)', +`DST_COUNTRY_CODE` UInt16 COMMENT 'Destination IP geo-country: two ASCII letters packed into a UInt16 (high byte = first letter)', +`SRC_LABEL` String COMMENT 'Resolved hostname or user-defined label for the source host', +`DST_LABEL` String COMMENT 'Resolved hostname or user-defined label for the destination host', +`SRC_MAC` UInt64 COMMENT 'Source MAC address encoded as a 64-bit integer', +`DST_MAC` UInt64 COMMENT 'Destination MAC address encoded as a 64-bit integer', +`COMMUNITY_ID` String COMMENT 'Community ID v1 flow hash for cross-tool correlation', +`SRC_ASN` UInt32 COMMENT 'Autonomous System Number of the source IP', +`DST_ASN` UInt32 COMMENT 'Autonomous System Number of the destination IP', +`PROBE_IP` UInt32 /* EXPORTER_IPV4_ADDRESS */ COMMENT 'IPv4 address of the NetFlow/IPFIX exporter (probe); same as EXPORTER_IPV4_ADDRESS', +`EXPORTER_SITE` UInt16 COMMENT 'Site/location identifier of the flow exporter', +`INTERFACE_ROLE` UInt8 COMMENT 'Role of the capturing interface (e.g. LAN, WAN)', +`OBSERVATION_POINT_ID` UInt16 COMMENT 'IPFIX observation point identifier', +`SRC2DST_TCP_FLAGS` UInt8 COMMENT 'Bitwise OR of TCP flags seen in the client-to-server direction', +`DST2SRC_TCP_FLAGS` UInt8 COMMENT 'Bitwise OR of TCP flags seen in the server-to-client direction', +`SCORE` UInt16 COMMENT 'Composite flow risk/security score', +`QOE_SCORE` UInt8 COMMENT 'Quality of Experience score (0=best)', +`CLIENT_NW_LATENCY_US` UInt32 COMMENT 'Estimated client-side network RTT in microseconds', +`SERVER_NW_LATENCY_US` UInt32 COMMENT 'Estimated server-side network RTT in microseconds', +`CLIENT_LOCATION` UInt8 COMMENT 'Client host location type (local LAN, remote, etc.)', +`SERVER_LOCATION` UInt8 COMMENT 'Server host location type (local LAN, remote, etc.)', +`SRC_NETWORK_ID` UInt32 COMMENT 'ntopng local-network ID for the source IP (0 if not a known local network)', +`DST_NETWORK_ID` UInt32 COMMENT 'ntopng local-network ID for the destination IP (0 if not a known local network)', +`CLIENT_FINGERPRINT` String COMMENT 'TLS/QUIC client fingerprint (JA3 or similar)', +`TCP_FINGERPRINT` String COMMENT 'TCP stack fingerprint used for passive OS detection', +`INPUT_SNMP` UInt32 COMMENT 'SNMP input interface index exported via NetFlow/IPFIX', +`OUTPUT_SNMP` UInt32 COMMENT 'SNMP output interface index exported via NetFlow/IPFIX', +`SRC_HOST_POOL_ID` UInt16 COMMENT 'ntopng host-pool ID of the source host', +`DST_HOST_POOL_ID` UInt16 COMMENT 'ntopng host-pool ID of the destination host', +`SRC_PROC_NAME` String COMMENT 'Name of the OS process that originated the flow (from eBPF/sysdig)', +`DST_PROC_NAME` String COMMENT 'Name of the OS process that received the flow (from eBPF/sysdig)', +`SRC_PROC_USER_NAME` String COMMENT 'OS username owning the source process', +`DST_PROC_USER_NAME` String COMMENT 'OS username owning the destination process', +`ALERTS_MAP` String COMMENT 'Serialized bitmap of individual alert conditions triggered on this flow', +`SEVERITY` UInt8 COMMENT 'Alert severity level; meaningful only when STATUS != 0', +`IS_CLI_ATTACKER` UInt8 COMMENT '1 if the client host is flagged as an attacker, 0 otherwise', +`IS_CLI_VICTIM` UInt8 COMMENT '1 if the client host is flagged as a victim, 0 otherwise', +`IS_CLI_BLACKLISTED` UInt8 COMMENT '1 if the client IP appears on a threat-intel blacklist, 0 otherwise', +`IS_SRV_ATTACKER` UInt8 COMMENT '1 if the server host is flagged as an attacker, 0 otherwise', +`IS_SRV_VICTIM` UInt8 COMMENT '1 if the server host is flagged as a victim, 0 otherwise', +`IS_SRV_BLACKLISTED` UInt8 COMMENT '1 if the server IP appears on a threat-intel blacklist, 0 otherwise', +`ALERT_STATUS` UInt8 COMMENT 'Alert lifecycle status (e.g. acknowledged, in-progress)', +`USER_LABEL` String COMMENT 'User-defined free-text label applied to this flow', +`USER_LABEL_TSTAMP` DateTime COMMENT 'Timestamp when USER_LABEL was last modified', +`PROTOCOL_INFO_JSON` String COMMENT 'Protocol-specific metadata (HTTP URL, DNS answers, TLS cert info, etc.) as JSON', +`ALERT_JSON` String COMMENT 'Alert-specific context and evidence as a JSON blob', +`IS_ALERT_DELETED` UInt8 COMMENT '1 if the alert on this flow was manually acknowledged/deleted, 0 otherwise', +`SRC2DST_PACKETS` UInt32 COMMENT 'Packet count from client (source) to server (destination)', +`DST2SRC_PACKETS` UInt32 COMMENT 'Packet count from server (destination) to client (source)', +`ALERT_CATEGORY` UInt8 COMMENT 'Alert category identifier (maps to ntopng AlertCategory enum)', +`MINOR_CONNECTION_STATE` UInt8 COMMENT 'Fine-grained TCP/flow connection state', +`MAJOR_CONNECTION_STATE` UInt8 COMMENT 'Coarse TCP connection state (e.g. established, closing, closed)', +`POST_NAT_IPV4_SRC_ADDR` UInt32 COMMENT 'Source IPv4 address after NAT translation', +`POST_NAT_SRC_PORT` UInt32 COMMENT 'Source port after NAT translation', +`POST_NAT_IPV4_DST_ADDR` UInt32 COMMENT 'Destination IPv4 address after NAT translation', +`POST_NAT_DST_PORT` UInt32 COMMENT 'Destination port after NAT translation', +`WLAN_SSID` String COMMENT 'Wireless LAN SSID associated with this flow', +`WTP_MAC_ADDRESS` UInt64 COMMENT 'MAC address of the Wireless Termination Point (access point) as a 64-bit integer', +`DOMAIN_NAME` String COMMENT 'Domain name extracted from the flow (from SNI, DNS, or HTTP Host header)', +`SRC_PEER_ASN` UInt32 COMMENT 'BGP peer ASN upstream of the source IP', +`DST_PEER_ASN` UInt32 COMMENT 'BGP peer ASN upstream of the destination IP', +`REQUIRE_ATTENTION` Boolean COMMENT 'True if this flow/alert has been flagged as requiring manual review' +) ENGINE = MergeTree() PARTITION BY toYYYYMMDD(FIRST_SEEN) ORDER BY (FIRST_SEEN, IPV4_SRC_ADDR, IPV4_DST_ADDR) +COMMENT 'Per-flow telemetry records captured locally or received via NetFlow/sFlow/IPFIX. Each row represents one bidirectional network flow with 5-tuple (src/dst IP, src/dst port, protocol), byte/packet counters, L7 application identification, flow-risk bitmap, DSCP, NAT addresses, process info, and optional alert metadata. Partitioned by day on FIRST_SEEN.'; @ ALTER TABLE flows ADD COLUMN IF NOT EXISTS `FLOW_ID` UInt64; @ @@ -202,27 +204,28 @@ ALTER TABLE flows DROP COLUMN IF EXISTS `PRE_NAT_DST_PORT`; @ CREATE TABLE IF NOT EXISTS `active_monitoring_alerts` ( -`rowid` UUID, -`alert_id` UInt32, -`alert_status` UInt8, -`interface_id` UInt16 DEFAULT 65535, -`resolved_ip` String, -`resolved_name` String, -`measurement` String, -`measure_threshold` UInt32 DEFAULT 0, -`measure_value` REAL DEFAULT 0.0, -`tstamp` DateTime, -`tstamp_end` DateTime DEFAULT toDateTime(0), -`severity` UInt8, -`score` UInt16, -`counter` UInt32, -`description` String, -`json` String, -`user_label` String, -`user_label_tstamp` DateTime DEFAULT toDateTime(0), -`alert_category` UInt8, -`require_attention` Boolean -) ENGINE = MergeTree() PARTITION BY toYYYYMMDD(tstamp) ORDER BY (tstamp); +`rowid` UUID COMMENT 'Unique identifier for this alert row (UUID v4)', +`alert_id` UInt32 COMMENT 'Alert type identifier (maps to ntopng alert type enum)', +`alert_status` UInt8 COMMENT 'Alert lifecycle status (0=engaged/active, 1=released/archived)', +`interface_id` UInt16 DEFAULT 65535 COMMENT 'ntopng interface identifier; 65535 means system/global scope', +`resolved_ip` String COMMENT 'IP address resolved from the monitored hostname at time of check', +`resolved_name` String COMMENT 'Hostname or target being monitored (FQDN or IP)', +`measurement` String COMMENT 'Type of active monitoring check (e.g. icmp, http, https, tls)', +`measure_threshold` UInt32 DEFAULT 0 COMMENT 'Configured threshold value that was exceeded to trigger the alert', +`measure_value` REAL DEFAULT 0.0 COMMENT 'Measured value (e.g. latency in ms, HTTP response code) at alert time', +`tstamp` DateTime COMMENT 'Timestamp when the alert was first triggered (alert start time)', +`tstamp_end` DateTime DEFAULT toDateTime(0) COMMENT 'Timestamp when the alert was resolved (zero/epoch if still active)', +`severity` UInt8 COMMENT 'Alert severity level (maps to ntopng AlertLevel enum)', +`score` UInt16 COMMENT 'Numeric risk/impact score associated with this alert', +`counter` UInt32 COMMENT 'Number of consecutive intervals this alert condition has been detected', +`description` String COMMENT 'Human-readable description of the alert', +`json` String COMMENT 'Additional alert context and metadata as a JSON blob', +`user_label` String COMMENT 'User-defined free-text label applied to this alert', +`user_label_tstamp` DateTime DEFAULT toDateTime(0) COMMENT 'Timestamp when user_label was last set', +`alert_category` UInt8 COMMENT 'Alert category (maps to ntopng AlertCategory enum)', +`require_attention` Boolean COMMENT 'True if this alert has been flagged as requiring manual attention' +) ENGINE = MergeTree() PARTITION BY toYYYYMMDD(tstamp) ORDER BY (tstamp) +COMMENT 'Historical alerts generated by the active monitoring subsystem (ICMP ping, HTTP, TLS checks, etc.). Rows are appended when an engaged alert is archived. See engaged_active_monitoring_alerts for currently-firing alerts and active_monitoring_alerts_view to query both together.'; @ ALTER TABLE `active_monitoring_alerts` ADD COLUMN IF NOT EXISTS alert_category UInt8; @ @@ -233,59 +236,61 @@ ALTER TABLE `active_monitoring_alerts` ADD COLUMN IF NOT EXISTS require_attentio DROP TABLE IF EXISTS `engaged_active_monitoring_alerts`; @ CREATE TABLE `engaged_active_monitoring_alerts` ( -`rowid` UUID, -`alert_id` UInt32, -`alert_status` UInt8, -`interface_id` UInt16 DEFAULT 65535, -`resolved_ip` String, -`resolved_name` String, -`measurement` String, -`measure_threshold` UInt32 DEFAULT 0, -`measure_value` REAL DEFAULT 0.0, -`tstamp` DateTime, -`tstamp_end` DateTime DEFAULT toDateTime(0), -`severity` UInt8, -`score` UInt16, -`counter` UInt32, -`description` String, -`json` String, -`user_label` String, -`user_label_tstamp` DateTime DEFAULT toDateTime(0), -`alert_category` UInt8, -`require_attention` Boolean -) ENGINE = Memory; +`rowid` UUID COMMENT 'Unique identifier for this alert row (UUID v4)', +`alert_id` UInt32 COMMENT 'Alert type identifier (maps to ntopng alert type enum)', +`alert_status` UInt8 COMMENT 'Alert lifecycle status (0=engaged/active, 1=released/archived)', +`interface_id` UInt16 DEFAULT 65535 COMMENT 'ntopng interface identifier; 65535 means system/global scope', +`resolved_ip` String COMMENT 'IP address resolved from the monitored hostname at time of check', +`resolved_name` String COMMENT 'Hostname or target being monitored (FQDN or IP)', +`measurement` String COMMENT 'Type of active monitoring check (e.g. icmp, http, https, tls)', +`measure_threshold` UInt32 DEFAULT 0 COMMENT 'Configured threshold value that was exceeded to trigger the alert', +`measure_value` REAL DEFAULT 0.0 COMMENT 'Measured value (e.g. latency in ms, HTTP response code) at alert time', +`tstamp` DateTime COMMENT 'Timestamp when the alert was first triggered (alert start time)', +`tstamp_end` DateTime DEFAULT toDateTime(0) COMMENT 'Timestamp when the alert was resolved (zero/epoch if still active)', +`severity` UInt8 COMMENT 'Alert severity level (maps to ntopng AlertLevel enum)', +`score` UInt16 COMMENT 'Numeric risk/impact score associated with this alert', +`counter` UInt32 COMMENT 'Number of consecutive intervals this alert condition has been detected', +`description` String COMMENT 'Human-readable description of the alert', +`json` String COMMENT 'Additional alert context and metadata as a JSON blob', +`user_label` String COMMENT 'User-defined free-text label applied to this alert', +`user_label_tstamp` DateTime DEFAULT toDateTime(0) COMMENT 'Timestamp when user_label was last set', +`alert_category` UInt8 COMMENT 'Alert category (maps to ntopng AlertCategory enum)', +`require_attention` Boolean COMMENT 'True if this alert has been flagged as requiring manual attention' +) ENGINE = Memory +COMMENT 'In-memory table holding currently active (engaged) active-monitoring alerts. Rows are inserted when an alert fires and removed when resolved. Merged with active_monitoring_alerts in active_monitoring_alerts_view.'; @ CREATE TABLE IF NOT EXISTS `host_alerts` ( -`rowid` UUID, -`alert_id` UInt32, -`alert_status` UInt8, -`interface_id` UInt16 DEFAULT 65535, -`ip_version` UInt8, -`ip` String, -`vlan_id` UInt16, -`name` String, -`is_attacker` UInt8, -`is_victim` UInt8, -`is_client` UInt8, -`is_server` UInt8, -`tstamp` DateTime, -`tstamp_end` DateTime, -`severity` UInt8, -`score` UInt16, -`granularity` UInt8, -`counter` UInt32, -`description` String, -`json` String, -`user_label` String, -`user_label_tstamp` DateTime, -`host_pool_id` UInt16, -`network` UInt16, -`country` String, -`alert_category` UInt8, -`require_attention` Boolean -) ENGINE = MergeTree() PARTITION BY toYYYYMMDD(tstamp) ORDER BY (tstamp); +`rowid` UUID COMMENT 'Unique identifier for this alert row (UUID v4)', +`alert_id` UInt32 COMMENT 'Alert type identifier (maps to ntopng alert type enum)', +`alert_status` UInt8 COMMENT 'Alert lifecycle status (0=engaged/active, 1=released/archived)', +`interface_id` UInt16 DEFAULT 65535 COMMENT 'ntopng interface identifier; 65535 means system/global scope', +`ip_version` UInt8 COMMENT 'IP version of the host: 4 for IPv4, 6 for IPv6', +`ip` String COMMENT 'IP address of the host that triggered the alert', +`vlan_id` UInt16 COMMENT 'VLAN on which the host was observed (0 if untagged)', +`name` String COMMENT 'Resolved hostname or user-defined name for the host', +`is_attacker` UInt8 COMMENT '1 if the host is the attacking party in this alert', +`is_victim` UInt8 COMMENT '1 if the host is the victim party in this alert', +`is_client` UInt8 COMMENT '1 if the host acted as a client in the triggering flow', +`is_server` UInt8 COMMENT '1 if the host acted as a server in the triggering flow', +`tstamp` DateTime COMMENT 'Timestamp when the alert was first triggered (alert start time)', +`tstamp_end` DateTime COMMENT 'Timestamp when the alert was resolved (zero/epoch if still active)', +`severity` UInt8 COMMENT 'Alert severity level (maps to ntopng AlertLevel enum)', +`score` UInt16 COMMENT 'Numeric risk/impact score associated with this alert', +`granularity` UInt8 COMMENT 'Periodic check interval that triggered the alert (e.g. 1=1min, 2=5min)', +`counter` UInt32 COMMENT 'Number of consecutive intervals this alert condition has been detected', +`description` String COMMENT 'Human-readable description of the alert', +`json` String COMMENT 'Additional alert context and metadata as a JSON blob', +`user_label` String COMMENT 'User-defined free-text label applied to this alert', +`user_label_tstamp` DateTime COMMENT 'Timestamp when user_label was last set', +`host_pool_id` UInt16 COMMENT 'ntopng host-pool ID the host belongs to', +`network` UInt16 COMMENT 'ntopng local-network ID the host belongs to', +`country` String COMMENT 'Two-letter ISO 3166-1 country code derived from the host IP', +`alert_category` UInt8 COMMENT 'Alert category (maps to ntopng AlertCategory enum)', +`require_attention` Boolean COMMENT 'True if this alert has been flagged as requiring manual attention' +) ENGINE = MergeTree() PARTITION BY toYYYYMMDD(tstamp) ORDER BY (tstamp) +COMMENT 'Historical alerts associated with individual hosts (identified by IP address and VLAN). Rows are appended when an engaged host alert is archived. See engaged_host_alerts for currently-firing alerts and host_alerts_view to query both together (with MITRE ATT&CK enrichment).'; @ ALTER TABLE `host_alerts` ADD COLUMN IF NOT EXISTS `host_pool_id` UInt16; @ @@ -302,60 +307,62 @@ ALTER TABLE `host_alerts` ADD COLUMN IF NOT EXISTS `require_attention` UInt8; DROP TABLE IF EXISTS `engaged_host_alerts`; @ CREATE TABLE `engaged_host_alerts` ( -`rowid` UUID, -`alert_id` UInt32, -`alert_status` UInt8, -`interface_id` UInt16 DEFAULT 65535, -`ip_version` UInt8, -`ip` String, -`vlan_id` UInt16, -`name` String, -`is_attacker` UInt8, -`is_victim` UInt8, -`is_client` UInt8, -`is_server` UInt8, -`tstamp` DateTime, -`tstamp_end` DateTime, -`severity` UInt8, -`score` UInt16, -`granularity` UInt8, -`counter` UInt32, -`description` String, -`json` String, -`user_label` String, -`user_label_tstamp` DateTime, -`host_pool_id` UInt16, -`network` UInt16, -`country` String, -`alert_category` UInt8, -`require_attention` Boolean -) ENGINE = Memory; +`rowid` UUID COMMENT 'Unique identifier for this alert row (UUID v4)', +`alert_id` UInt32 COMMENT 'Alert type identifier (maps to ntopng alert type enum)', +`alert_status` UInt8 COMMENT 'Alert lifecycle status (0=engaged/active, 1=released/archived)', +`interface_id` UInt16 DEFAULT 65535 COMMENT 'ntopng interface identifier; 65535 means system/global scope', +`ip_version` UInt8 COMMENT 'IP version of the host: 4 for IPv4, 6 for IPv6', +`ip` String COMMENT 'IP address of the host that triggered the alert', +`vlan_id` UInt16 COMMENT 'VLAN on which the host was observed (0 if untagged)', +`name` String COMMENT 'Resolved hostname or user-defined name for the host', +`is_attacker` UInt8 COMMENT '1 if the host is the attacking party in this alert', +`is_victim` UInt8 COMMENT '1 if the host is the victim party in this alert', +`is_client` UInt8 COMMENT '1 if the host acted as a client in the triggering flow', +`is_server` UInt8 COMMENT '1 if the host acted as a server in the triggering flow', +`tstamp` DateTime COMMENT 'Timestamp when the alert was first triggered (alert start time)', +`tstamp_end` DateTime COMMENT 'Timestamp when the alert was resolved (zero/epoch if still active)', +`severity` UInt8 COMMENT 'Alert severity level (maps to ntopng AlertLevel enum)', +`score` UInt16 COMMENT 'Numeric risk/impact score associated with this alert', +`granularity` UInt8 COMMENT 'Periodic check interval that triggered the alert (e.g. 1=1min, 2=5min)', +`counter` UInt32 COMMENT 'Number of consecutive intervals this alert condition has been detected', +`description` String COMMENT 'Human-readable description of the alert', +`json` String COMMENT 'Additional alert context and metadata as a JSON blob', +`user_label` String COMMENT 'User-defined free-text label applied to this alert', +`user_label_tstamp` DateTime COMMENT 'Timestamp when user_label was last set', +`host_pool_id` UInt16 COMMENT 'ntopng host-pool ID the host belongs to', +`network` UInt16 COMMENT 'ntopng local-network ID the host belongs to', +`country` String COMMENT 'Two-letter ISO 3166-1 country code derived from the host IP', +`alert_category` UInt8 COMMENT 'Alert category (maps to ntopng AlertCategory enum)', +`require_attention` Boolean COMMENT 'True if this alert has been flagged as requiring manual attention' +) ENGINE = Memory +COMMENT 'In-memory table holding currently active (engaged) host alerts. Rows are inserted when an alert fires and removed when resolved. Merged with host_alerts in host_alerts_view.'; @ CREATE TABLE IF NOT EXISTS `mac_alerts` ( -`rowid` UUID, -`alert_id` UInt32, -`alert_status` UInt8, -`interface_id` UInt16 DEFAULT 65535, -`address` String, -`device_type` UInt8 DEFAULT 0, -`name` String, -`is_attacker` UInt8, -`is_victim` UInt8, -`tstamp` DateTime, -`tstamp_end` DateTime, -`severity` UInt8, -`score` UInt16, -`granularity` UInt8, -`counter` UInt32, -`description` String, -`json` String, -`user_label` String, -`user_label_tstamp` DateTime, -`alert_category` UInt8, -`require_attention` Boolean -) ENGINE = MergeTree() PARTITION BY toYYYYMMDD(tstamp) ORDER BY (tstamp); +`rowid` UUID COMMENT 'Unique identifier for this alert row (UUID v4)', +`alert_id` UInt32 COMMENT 'Alert type identifier (maps to ntopng alert type enum)', +`alert_status` UInt8 COMMENT 'Alert lifecycle status (0=engaged/active, 1=released/archived)', +`interface_id` UInt16 DEFAULT 65535 COMMENT 'ntopng interface identifier; 65535 means system/global scope', +`address` String COMMENT 'MAC address of the device that triggered the alert (colon-separated hex)', +`device_type` UInt8 DEFAULT 0 COMMENT 'Device category/type identifier (maps to ntopng DeviceType enum)', +`name` String COMMENT 'User-defined or discovered name for the device', +`is_attacker` UInt8 COMMENT '1 if the device is the attacking party in this alert', +`is_victim` UInt8 COMMENT '1 if the device is the victim party in this alert', +`tstamp` DateTime COMMENT 'Timestamp when the alert was first triggered (alert start time)', +`tstamp_end` DateTime COMMENT 'Timestamp when the alert was resolved (zero/epoch if still active)', +`severity` UInt8 COMMENT 'Alert severity level (maps to ntopng AlertLevel enum)', +`score` UInt16 COMMENT 'Numeric risk/impact score associated with this alert', +`granularity` UInt8 COMMENT 'Periodic check interval that triggered the alert (e.g. 1=1min, 2=5min)', +`counter` UInt32 COMMENT 'Number of consecutive intervals this alert condition has been detected', +`description` String COMMENT 'Human-readable description of the alert', +`json` String COMMENT 'Additional alert context and metadata as a JSON blob', +`user_label` String COMMENT 'User-defined free-text label applied to this alert', +`user_label_tstamp` DateTime COMMENT 'Timestamp when user_label was last set', +`alert_category` UInt8 COMMENT 'Alert category (maps to ntopng AlertCategory enum)', +`require_attention` Boolean COMMENT 'True if this alert has been flagged as requiring manual attention' +) ENGINE = MergeTree() PARTITION BY toYYYYMMDD(tstamp) ORDER BY (tstamp) +COMMENT 'Historical alerts associated with MAC addresses and layer-2 devices. See engaged_mac_alerts for currently-firing alerts and mac_alerts_view to query both together.'; @ ALTER TABLE `mac_alerts` ADD COLUMN IF NOT EXISTS alert_category UInt8; @ @@ -366,53 +373,55 @@ ALTER TABLE `mac_alerts` ADD COLUMN IF NOT EXISTS require_attention Boolean; DROP TABLE IF EXISTS `engaged_mac_alerts`; @ CREATE TABLE `engaged_mac_alerts` ( -`rowid` UUID, -`alert_id` UInt32, -`alert_status` UInt8, -`interface_id` UInt16 DEFAULT 65535, -`address` String, -`device_type` UInt8 DEFAULT 0, -`name` String, -`is_attacker` UInt8, -`is_victim` UInt8, -`tstamp` DateTime, -`tstamp_end` DateTime, -`severity` UInt8, -`score` UInt16, -`granularity` UInt8, -`counter` UInt32, -`description` String, -`json` String, -`user_label` String, -`user_label_tstamp` DateTime, -`alert_category` UInt8, -`require_attention` Boolean -) ENGINE = Memory; +`rowid` UUID COMMENT 'Unique identifier for this alert row (UUID v4)', +`alert_id` UInt32 COMMENT 'Alert type identifier (maps to ntopng alert type enum)', +`alert_status` UInt8 COMMENT 'Alert lifecycle status (0=engaged/active, 1=released/archived)', +`interface_id` UInt16 DEFAULT 65535 COMMENT 'ntopng interface identifier; 65535 means system/global scope', +`address` String COMMENT 'MAC address of the device that triggered the alert (colon-separated hex)', +`device_type` UInt8 DEFAULT 0 COMMENT 'Device category/type identifier (maps to ntopng DeviceType enum)', +`name` String COMMENT 'User-defined or discovered name for the device', +`is_attacker` UInt8 COMMENT '1 if the device is the attacking party in this alert', +`is_victim` UInt8 COMMENT '1 if the device is the victim party in this alert', +`tstamp` DateTime COMMENT 'Timestamp when the alert was first triggered (alert start time)', +`tstamp_end` DateTime COMMENT 'Timestamp when the alert was resolved (zero/epoch if still active)', +`severity` UInt8 COMMENT 'Alert severity level (maps to ntopng AlertLevel enum)', +`score` UInt16 COMMENT 'Numeric risk/impact score associated with this alert', +`granularity` UInt8 COMMENT 'Periodic check interval that triggered the alert (e.g. 1=1min, 2=5min)', +`counter` UInt32 COMMENT 'Number of consecutive intervals this alert condition has been detected', +`description` String COMMENT 'Human-readable description of the alert', +`json` String COMMENT 'Additional alert context and metadata as a JSON blob', +`user_label` String COMMENT 'User-defined free-text label applied to this alert', +`user_label_tstamp` DateTime COMMENT 'Timestamp when user_label was last set', +`alert_category` UInt8 COMMENT 'Alert category (maps to ntopng AlertCategory enum)', +`require_attention` Boolean COMMENT 'True if this alert has been flagged as requiring manual attention' +) ENGINE = Memory +COMMENT 'In-memory table holding currently active (engaged) MAC/device alerts. Rows are inserted when an alert fires and removed when resolved. Merged with mac_alerts in mac_alerts_view.'; @ CREATE TABLE IF NOT EXISTS `snmp_alerts` ( -`rowid` UUID, -`alert_id` UInt32, -`alert_status` UInt8, -`interface_id` UInt16 DEFAULT 65535, -`ip` String, -`port` UInt32, -`name` String, -`port_name` String, -`tstamp` DateTime, -`tstamp_end` DateTime, -`severity` UInt8, -`score` UInt16, -`granularity` UInt8, -`counter` UInt32, -`description` String, -`json` String, -`user_label` String, -`user_label_tstamp` DateTime, -`alert_category` UInt8, -`require_attention` Boolean -) ENGINE = MergeTree() PARTITION BY toYYYYMMDD(tstamp) ORDER BY (tstamp); +`rowid` UUID COMMENT 'Unique identifier for this alert row (UUID v4)', +`alert_id` UInt32 COMMENT 'Alert type identifier (maps to ntopng alert type enum)', +`alert_status` UInt8 COMMENT 'Alert lifecycle status (0=engaged/active, 1=released/archived)', +`interface_id` UInt16 DEFAULT 65535 COMMENT 'ntopng interface identifier; 65535 means system/global scope', +`ip` String COMMENT 'IP address of the SNMP-polled device that triggered the alert', +`port` UInt32 COMMENT 'SNMP interface index (ifIndex) of the interface that triggered the alert', +`name` String COMMENT 'SNMP sysName or user-defined name of the device', +`port_name` String COMMENT 'SNMP ifDescr or user-defined name of the triggering interface', +`tstamp` DateTime COMMENT 'Timestamp when the alert was first triggered (alert start time)', +`tstamp_end` DateTime COMMENT 'Timestamp when the alert was resolved (zero/epoch if still active)', +`severity` UInt8 COMMENT 'Alert severity level (maps to ntopng AlertLevel enum)', +`score` UInt16 COMMENT 'Numeric risk/impact score associated with this alert', +`granularity` UInt8 COMMENT 'Periodic check interval that triggered the alert (e.g. 1=1min, 2=5min)', +`counter` UInt32 COMMENT 'Number of consecutive intervals this alert condition has been detected', +`description` String COMMENT 'Human-readable description of the alert', +`json` String COMMENT 'Additional alert context and metadata as a JSON blob', +`user_label` String COMMENT 'User-defined free-text label applied to this alert', +`user_label_tstamp` DateTime COMMENT 'Timestamp when user_label was last set', +`alert_category` UInt8 COMMENT 'Alert category (maps to ntopng AlertCategory enum)', +`require_attention` Boolean COMMENT 'True if this alert has been flagged as requiring manual attention' +) ENGINE = MergeTree() PARTITION BY toYYYYMMDD(tstamp) ORDER BY (tstamp) +COMMENT 'Historical alerts from SNMP-polled network devices and their individual ports. See engaged_snmp_alerts for currently-firing alerts and snmp_alerts_view to query both together.'; @ ALTER TABLE `snmp_alerts` MODIFY COLUMN `port` UInt32; @ @@ -425,51 +434,53 @@ ALTER TABLE `snmp_alerts` ADD COLUMN IF NOT EXISTS require_attention Boolean; DROP TABLE IF EXISTS `engaged_snmp_alerts`; @ CREATE TABLE `engaged_snmp_alerts` ( -`rowid` UUID, -`alert_id` UInt32, -`alert_status` UInt8, -`interface_id` UInt16 DEFAULT 65535, -`ip` String, -`port` UInt32, -`name` String, -`port_name` String, -`tstamp` DateTime, -`tstamp_end` DateTime, -`severity` UInt8, -`score` UInt16, -`granularity` UInt8, -`counter` UInt32, -`description` String, -`json` String, -`user_label` String, -`user_label_tstamp` DateTime, -`alert_category` UInt8, -`require_attention` Boolean -) ENGINE = Memory; +`rowid` UUID COMMENT 'Unique identifier for this alert row (UUID v4)', +`alert_id` UInt32 COMMENT 'Alert type identifier (maps to ntopng alert type enum)', +`alert_status` UInt8 COMMENT 'Alert lifecycle status (0=engaged/active, 1=released/archived)', +`interface_id` UInt16 DEFAULT 65535 COMMENT 'ntopng interface identifier; 65535 means system/global scope', +`ip` String COMMENT 'IP address of the SNMP-polled device that triggered the alert', +`port` UInt32 COMMENT 'SNMP interface index (ifIndex) of the interface that triggered the alert', +`name` String COMMENT 'SNMP sysName or user-defined name of the device', +`port_name` String COMMENT 'SNMP ifDescr or user-defined name of the triggering interface', +`tstamp` DateTime COMMENT 'Timestamp when the alert was first triggered (alert start time)', +`tstamp_end` DateTime COMMENT 'Timestamp when the alert was resolved (zero/epoch if still active)', +`severity` UInt8 COMMENT 'Alert severity level (maps to ntopng AlertLevel enum)', +`score` UInt16 COMMENT 'Numeric risk/impact score associated with this alert', +`granularity` UInt8 COMMENT 'Periodic check interval that triggered the alert (e.g. 1=1min, 2=5min)', +`counter` UInt32 COMMENT 'Number of consecutive intervals this alert condition has been detected', +`description` String COMMENT 'Human-readable description of the alert', +`json` String COMMENT 'Additional alert context and metadata as a JSON blob', +`user_label` String COMMENT 'User-defined free-text label applied to this alert', +`user_label_tstamp` DateTime COMMENT 'Timestamp when user_label was last set', +`alert_category` UInt8 COMMENT 'Alert category (maps to ntopng AlertCategory enum)', +`require_attention` Boolean COMMENT 'True if this alert has been flagged as requiring manual attention' +) ENGINE = Memory +COMMENT 'In-memory table holding currently active (engaged) SNMP alerts. Rows are inserted when an alert fires and removed when resolved. Merged with snmp_alerts in snmp_alerts_view.'; @ CREATE TABLE IF NOT EXISTS `network_alerts` ( -`rowid` UUID, -`local_network_id` UInt16, -`alert_id` UInt32, -`alert_status` UInt8, -`interface_id` UInt16 DEFAULT 65535, -`name` String, -`alias` String, -`tstamp` DateTime, -`tstamp_end` DateTime, -`severity` UInt8, -`score` UInt16, -`granularity` UInt8, -`counter` UInt32, -`description` String, -`json` String, -`user_label` String, -`user_label_tstamp` DateTime, -`alert_category` UInt8, -`require_attention` Boolean -) ENGINE = MergeTree() PARTITION BY toYYYYMMDD(tstamp) ORDER BY (tstamp); +`rowid` UUID COMMENT 'Unique identifier for this alert row (UUID v4)', +`local_network_id` UInt16 COMMENT 'ntopng internal identifier of the local network subnet', +`alert_id` UInt32 COMMENT 'Alert type identifier (maps to ntopng alert type enum)', +`alert_status` UInt8 COMMENT 'Alert lifecycle status (0=engaged/active, 1=released/archived)', +`interface_id` UInt16 DEFAULT 65535 COMMENT 'ntopng interface identifier; 65535 means system/global scope', +`name` String COMMENT 'CIDR notation or user-defined name of the network (e.g. 192.168.1.0/24)', +`alias` String COMMENT 'User-defined alias for the network', +`tstamp` DateTime COMMENT 'Timestamp when the alert was first triggered (alert start time)', +`tstamp_end` DateTime COMMENT 'Timestamp when the alert was resolved (zero/epoch if still active)', +`severity` UInt8 COMMENT 'Alert severity level (maps to ntopng AlertLevel enum)', +`score` UInt16 COMMENT 'Numeric risk/impact score associated with this alert', +`granularity` UInt8 COMMENT 'Periodic check interval that triggered the alert (e.g. 1=1min, 2=5min)', +`counter` UInt32 COMMENT 'Number of consecutive intervals this alert condition has been detected', +`description` String COMMENT 'Human-readable description of the alert', +`json` String COMMENT 'Additional alert context and metadata as a JSON blob', +`user_label` String COMMENT 'User-defined free-text label applied to this alert', +`user_label_tstamp` DateTime COMMENT 'Timestamp when user_label was last set', +`alert_category` UInt8 COMMENT 'Alert category (maps to ntopng AlertCategory enum)', +`require_attention` Boolean COMMENT 'True if this alert has been flagged as requiring manual attention' +) ENGINE = MergeTree() PARTITION BY toYYYYMMDD(tstamp) ORDER BY (tstamp) +COMMENT 'Historical alerts associated with local network subnets (identified by local_network_id). See engaged_network_alerts for currently-firing alerts and network_alerts_view to query both together.'; @ ALTER TABLE `network_alerts` ADD COLUMN IF NOT EXISTS alert_category UInt8; @ @@ -480,101 +491,105 @@ ALTER TABLE `network_alerts` ADD COLUMN IF NOT EXISTS require_attention Boolean; DROP TABLE IF EXISTS `engaged_network_alerts`; @ CREATE TABLE `engaged_network_alerts` ( -`rowid` UUID, -`local_network_id` UInt16, -`alert_id` UInt32, -`alert_status` UInt8, -`interface_id` UInt16 DEFAULT 65535, -`name` String, -`alias` String, -`tstamp` DateTime, -`tstamp_end` DateTime, -`severity` UInt8, -`score` UInt16, -`granularity` UInt8, -`counter` UInt32, -`description` String, -`json` String, -`user_label` String, -`user_label_tstamp` DateTime, -`alert_category` UInt8, -`require_attention` Boolean -) ENGINE = Memory; +`rowid` UUID COMMENT 'Unique identifier for this alert row (UUID v4)', +`local_network_id` UInt16 COMMENT 'ntopng internal identifier of the local network subnet', +`alert_id` UInt32 COMMENT 'Alert type identifier (maps to ntopng alert type enum)', +`alert_status` UInt8 COMMENT 'Alert lifecycle status (0=engaged/active, 1=released/archived)', +`interface_id` UInt16 DEFAULT 65535 COMMENT 'ntopng interface identifier; 65535 means system/global scope', +`name` String COMMENT 'CIDR notation or user-defined name of the network (e.g. 192.168.1.0/24)', +`alias` String COMMENT 'User-defined alias for the network', +`tstamp` DateTime COMMENT 'Timestamp when the alert was first triggered (alert start time)', +`tstamp_end` DateTime COMMENT 'Timestamp when the alert was resolved (zero/epoch if still active)', +`severity` UInt8 COMMENT 'Alert severity level (maps to ntopng AlertLevel enum)', +`score` UInt16 COMMENT 'Numeric risk/impact score associated with this alert', +`granularity` UInt8 COMMENT 'Periodic check interval that triggered the alert (e.g. 1=1min, 2=5min)', +`counter` UInt32 COMMENT 'Number of consecutive intervals this alert condition has been detected', +`description` String COMMENT 'Human-readable description of the alert', +`json` String COMMENT 'Additional alert context and metadata as a JSON blob', +`user_label` String COMMENT 'User-defined free-text label applied to this alert', +`user_label_tstamp` DateTime COMMENT 'Timestamp when user_label was last set', +`alert_category` UInt8 COMMENT 'Alert category (maps to ntopng AlertCategory enum)', +`require_attention` Boolean COMMENT 'True if this alert has been flagged as requiring manual attention' +) ENGINE = Memory +COMMENT 'In-memory table holding currently active (engaged) network/subnet alerts. Rows are inserted when an alert fires and removed when resolved. Merged with network_alerts in network_alerts_view.'; @ CREATE TABLE IF NOT EXISTS `as_alerts` ( -`rowid` UUID, -`asn` UInt32, -`alert_id` UInt32, -`alert_status` UInt8, -`interface_id` UInt16 DEFAULT 65535, -`name` String, -`alias` String, -`tstamp` DateTime, -`tstamp_end` DateTime, -`severity` UInt8, -`score` UInt16, -`granularity` UInt8, -`counter` UInt32, -`description` String, -`json` String, -`user_label` String, -`user_label_tstamp` DateTime, -`alert_category` UInt8, -`require_attention` Boolean -) ENGINE = MergeTree() PARTITION BY toYYYYMMDD(tstamp) ORDER BY (tstamp); +`rowid` UUID COMMENT 'Unique identifier for this alert row (UUID v4)', +`asn` UInt32 COMMENT 'Autonomous System Number that triggered the alert', +`alert_id` UInt32 COMMENT 'Alert type identifier (maps to ntopng alert type enum)', +`alert_status` UInt8 COMMENT 'Alert lifecycle status (0=engaged/active, 1=released/archived)', +`interface_id` UInt16 DEFAULT 65535 COMMENT 'ntopng interface identifier; 65535 means system/global scope', +`name` String COMMENT 'AS name/description (from WHOIS or user configuration)', +`alias` String COMMENT 'User-defined alias for this AS', +`tstamp` DateTime COMMENT 'Timestamp when the alert was first triggered (alert start time)', +`tstamp_end` DateTime COMMENT 'Timestamp when the alert was resolved (zero/epoch if still active)', +`severity` UInt8 COMMENT 'Alert severity level (maps to ntopng AlertLevel enum)', +`score` UInt16 COMMENT 'Numeric risk/impact score associated with this alert', +`granularity` UInt8 COMMENT 'Periodic check interval that triggered the alert (e.g. 1=1min, 2=5min)', +`counter` UInt32 COMMENT 'Number of consecutive intervals this alert condition has been detected', +`description` String COMMENT 'Human-readable description of the alert', +`json` String COMMENT 'Additional alert context and metadata as a JSON blob', +`user_label` String COMMENT 'User-defined free-text label applied to this alert', +`user_label_tstamp` DateTime COMMENT 'Timestamp when user_label was last set', +`alert_category` UInt8 COMMENT 'Alert category (maps to ntopng AlertCategory enum)', +`require_attention` Boolean COMMENT 'True if this alert has been flagged as requiring manual attention' +) ENGINE = MergeTree() PARTITION BY toYYYYMMDD(tstamp) ORDER BY (tstamp) +COMMENT 'Historical alerts associated with Autonomous Systems (identified by ASN). See engaged_as_alerts for currently-firing alerts and as_alerts_view to query both together.'; @ DROP TABLE IF EXISTS `engaged_as_alerts`; @ CREATE TABLE `engaged_as_alerts` ( -`rowid` UUID, -`asn` UInt32, -`alert_id` UInt32, -`alert_status` UInt8, -`interface_id` UInt16 DEFAULT 65535, -`name` String, -`alias` String, -`tstamp` DateTime, -`tstamp_end` DateTime, -`severity` UInt8, -`score` UInt16, -`granularity` UInt8, -`counter` UInt32, -`description` String, -`json` String, -`user_label` String, -`user_label_tstamp` DateTime, -`alert_category` UInt8, -`require_attention` Boolean -) ENGINE = Memory; +`rowid` UUID COMMENT 'Unique identifier for this alert row (UUID v4)', +`asn` UInt32 COMMENT 'Autonomous System Number that triggered the alert', +`alert_id` UInt32 COMMENT 'Alert type identifier (maps to ntopng alert type enum)', +`alert_status` UInt8 COMMENT 'Alert lifecycle status (0=engaged/active, 1=released/archived)', +`interface_id` UInt16 DEFAULT 65535 COMMENT 'ntopng interface identifier; 65535 means system/global scope', +`name` String COMMENT 'AS name/description (from WHOIS or user configuration)', +`alias` String COMMENT 'User-defined alias for this AS', +`tstamp` DateTime COMMENT 'Timestamp when the alert was first triggered (alert start time)', +`tstamp_end` DateTime COMMENT 'Timestamp when the alert was resolved (zero/epoch if still active)', +`severity` UInt8 COMMENT 'Alert severity level (maps to ntopng AlertLevel enum)', +`score` UInt16 COMMENT 'Numeric risk/impact score associated with this alert', +`granularity` UInt8 COMMENT 'Periodic check interval that triggered the alert (e.g. 1=1min, 2=5min)', +`counter` UInt32 COMMENT 'Number of consecutive intervals this alert condition has been detected', +`description` String COMMENT 'Human-readable description of the alert', +`json` String COMMENT 'Additional alert context and metadata as a JSON blob', +`user_label` String COMMENT 'User-defined free-text label applied to this alert', +`user_label_tstamp` DateTime COMMENT 'Timestamp when user_label was last set', +`alert_category` UInt8 COMMENT 'Alert category (maps to ntopng AlertCategory enum)', +`require_attention` Boolean COMMENT 'True if this alert has been flagged as requiring manual attention' +) ENGINE = Memory +COMMENT 'In-memory table holding currently active (engaged) Autonomous System alerts. Rows are inserted when an alert fires and removed when resolved. Merged with as_alerts in as_alerts_view.'; @ CREATE TABLE IF NOT EXISTS `interface_alerts` ( -`rowid` UUID, -`ifid` UInt8, -`alert_id` UInt32, -`alert_status` UInt8, -`interface_id` UInt16 DEFAULT 65535, -`subtype` String, -`name` String, -`alias` String, -`tstamp` DateTime, -`tstamp_end` DateTime, -`severity` UInt8, -`score` UInt16, -`granularity` UInt8, -`counter` UInt32, -`description` String, -`json` String, -`user_label` String, -`user_label_tstamp` DateTime, -`alert_category` UInt8, -`require_attention` Boolean -) ENGINE = MergeTree() PARTITION BY toYYYYMMDD(tstamp) ORDER BY (tstamp); +`rowid` UUID COMMENT 'Unique identifier for this alert row (UUID v4)', +`ifid` UInt8 COMMENT 'ntopng internal interface index', +`alert_id` UInt32 COMMENT 'Alert type identifier (maps to ntopng alert type enum)', +`alert_status` UInt8 COMMENT 'Alert lifecycle status (0=engaged/active, 1=released/archived)', +`interface_id` UInt16 DEFAULT 65535 COMMENT 'ntopng interface identifier; 65535 means system/global scope', +`subtype` String COMMENT 'Alert sub-type string providing additional context', +`name` String COMMENT 'Interface name (e.g. eth0, wlan0)', +`alias` String COMMENT 'User-defined alias for the interface', +`tstamp` DateTime COMMENT 'Timestamp when the alert was first triggered (alert start time)', +`tstamp_end` DateTime COMMENT 'Timestamp when the alert was resolved (zero/epoch if still active)', +`severity` UInt8 COMMENT 'Alert severity level (maps to ntopng AlertLevel enum)', +`score` UInt16 COMMENT 'Numeric risk/impact score associated with this alert', +`granularity` UInt8 COMMENT 'Periodic check interval that triggered the alert (e.g. 1=1min, 2=5min)', +`counter` UInt32 COMMENT 'Number of consecutive intervals this alert condition has been detected', +`description` String COMMENT 'Human-readable description of the alert', +`json` String COMMENT 'Additional alert context and metadata as a JSON blob', +`user_label` String COMMENT 'User-defined free-text label applied to this alert', +`user_label_tstamp` DateTime COMMENT 'Timestamp when user_label was last set', +`alert_category` UInt8 COMMENT 'Alert category (maps to ntopng AlertCategory enum)', +`require_attention` Boolean COMMENT 'True if this alert has been flagged as requiring manual attention' +) ENGINE = MergeTree() PARTITION BY toYYYYMMDD(tstamp) ORDER BY (tstamp) +COMMENT 'Historical alerts associated with monitored network interfaces. See engaged_interface_alerts for currently-firing alerts and interface_alerts_view to query both together.'; @ ALTER TABLE `interface_alerts` ADD COLUMN IF NOT EXISTS alert_category UInt8; @ @@ -585,49 +600,51 @@ ALTER TABLE `interface_alerts` ADD COLUMN IF NOT EXISTS require_attention Boolea DROP TABLE IF EXISTS `engaged_interface_alerts`; @ CREATE TABLE `engaged_interface_alerts` ( -`rowid` UUID, -`ifid` UInt8, -`alert_id` UInt32, -`alert_status` UInt8, -`interface_id` UInt16 DEFAULT 65535, -`subtype` String, -`name` String, -`alias` String, -`tstamp` DateTime, -`tstamp_end` DateTime, -`severity` UInt8, -`score` UInt16, -`granularity` UInt8, -`counter` UInt32, -`description` String, -`json` String, -`user_label` String, -`user_label_tstamp` DateTime, -`alert_category` UInt8, -`require_attention` Boolean -) ENGINE = Memory; +`rowid` UUID COMMENT 'Unique identifier for this alert row (UUID v4)', +`ifid` UInt8 COMMENT 'ntopng internal interface index', +`alert_id` UInt32 COMMENT 'Alert type identifier (maps to ntopng alert type enum)', +`alert_status` UInt8 COMMENT 'Alert lifecycle status (0=engaged/active, 1=released/archived)', +`interface_id` UInt16 DEFAULT 65535 COMMENT 'ntopng interface identifier; 65535 means system/global scope', +`subtype` String COMMENT 'Alert sub-type string providing additional context', +`name` String COMMENT 'Interface name (e.g. eth0, wlan0)', +`alias` String COMMENT 'User-defined alias for the interface', +`tstamp` DateTime COMMENT 'Timestamp when the alert was first triggered (alert start time)', +`tstamp_end` DateTime COMMENT 'Timestamp when the alert was resolved (zero/epoch if still active)', +`severity` UInt8 COMMENT 'Alert severity level (maps to ntopng AlertLevel enum)', +`score` UInt16 COMMENT 'Numeric risk/impact score associated with this alert', +`granularity` UInt8 COMMENT 'Periodic check interval that triggered the alert (e.g. 1=1min, 2=5min)', +`counter` UInt32 COMMENT 'Number of consecutive intervals this alert condition has been detected', +`description` String COMMENT 'Human-readable description of the alert', +`json` String COMMENT 'Additional alert context and metadata as a JSON blob', +`user_label` String COMMENT 'User-defined free-text label applied to this alert', +`user_label_tstamp` DateTime COMMENT 'Timestamp when user_label was last set', +`alert_category` UInt8 COMMENT 'Alert category (maps to ntopng AlertCategory enum)', +`require_attention` Boolean COMMENT 'True if this alert has been flagged as requiring manual attention' +) ENGINE = Memory +COMMENT 'In-memory table holding currently active (engaged) interface alerts. Rows are inserted when an alert fires and removed when resolved. Merged with interface_alerts in interface_alerts_view.'; @ CREATE TABLE IF NOT EXISTS `user_alerts` ( -`rowid` UUID, -`alert_id` UInt32, -`alert_status` UInt8, -`interface_id` UInt16 DEFAULT 65535, -`user` String, -`tstamp` DateTime, -`tstamp_end` DateTime, -`severity` UInt8, -`score` UInt16, -`granularity` UInt8, -`counter` UInt32, -`description` String, -`json` String, -`user_label` String, -`user_label_tstamp` DateTime, -`alert_category` UInt8, -`require_attention` Boolean -) ENGINE = MergeTree() PARTITION BY toYYYYMMDD(tstamp) ORDER BY (tstamp); +`rowid` UUID COMMENT 'Unique identifier for this alert row (UUID v4)', +`alert_id` UInt32 COMMENT 'Alert type identifier (maps to ntopng alert type enum)', +`alert_status` UInt8 COMMENT 'Alert lifecycle status (0=engaged/active, 1=released/archived)', +`interface_id` UInt16 DEFAULT 65535 COMMENT 'ntopng interface identifier; 65535 means system/global scope', +`user` String COMMENT 'ntopng username associated with this alert', +`tstamp` DateTime COMMENT 'Timestamp when the alert was first triggered (alert start time)', +`tstamp_end` DateTime COMMENT 'Timestamp when the alert was resolved (zero/epoch if still active)', +`severity` UInt8 COMMENT 'Alert severity level (maps to ntopng AlertLevel enum)', +`score` UInt16 COMMENT 'Numeric risk/impact score associated with this alert', +`granularity` UInt8 COMMENT 'Periodic check interval that triggered the alert (e.g. 1=1min, 2=5min)', +`counter` UInt32 COMMENT 'Number of consecutive intervals this alert condition has been detected', +`description` String COMMENT 'Human-readable description of the alert', +`json` String COMMENT 'Additional alert context and metadata as a JSON blob', +`user_label` String COMMENT 'User-defined free-text label applied to this alert', +`user_label_tstamp` DateTime COMMENT 'Timestamp when user_label was last set', +`alert_category` UInt8 COMMENT 'Alert category (maps to ntopng AlertCategory enum)', +`require_attention` Boolean COMMENT 'True if this alert has been flagged as requiring manual attention' +) ENGINE = MergeTree() PARTITION BY toYYYYMMDD(tstamp) ORDER BY (tstamp) +COMMENT 'Historical alerts associated with ntopng-managed users. See engaged_user_alerts for currently-firing alerts and user_alerts_view to query both together.'; @ ALTER TABLE `user_alerts` ADD COLUMN IF NOT EXISTS alert_category UInt8; @ @@ -638,46 +655,48 @@ ALTER TABLE `user_alerts` ADD COLUMN IF NOT EXISTS require_attention Boolean; DROP TABLE IF EXISTS `engaged_user_alerts`; @ CREATE TABLE `engaged_user_alerts` ( -`rowid` UUID, -`alert_id` UInt32, -`alert_status` UInt8, -`interface_id` UInt16 DEFAULT 65535, -`user` String, -`tstamp` DateTime, -`tstamp_end` DateTime, -`severity` UInt8, -`score` UInt16, -`granularity` UInt8, -`counter` UInt32, -`description` String, -`json` String, -`user_label` String, -`user_label_tstamp` DateTime, -`alert_category` UInt8, -`require_attention` Boolean -) ENGINE = Memory; +`rowid` UUID COMMENT 'Unique identifier for this alert row (UUID v4)', +`alert_id` UInt32 COMMENT 'Alert type identifier (maps to ntopng alert type enum)', +`alert_status` UInt8 COMMENT 'Alert lifecycle status (0=engaged/active, 1=released/archived)', +`interface_id` UInt16 DEFAULT 65535 COMMENT 'ntopng interface identifier; 65535 means system/global scope', +`user` String COMMENT 'ntopng username associated with this alert', +`tstamp` DateTime COMMENT 'Timestamp when the alert was first triggered (alert start time)', +`tstamp_end` DateTime COMMENT 'Timestamp when the alert was resolved (zero/epoch if still active)', +`severity` UInt8 COMMENT 'Alert severity level (maps to ntopng AlertLevel enum)', +`score` UInt16 COMMENT 'Numeric risk/impact score associated with this alert', +`granularity` UInt8 COMMENT 'Periodic check interval that triggered the alert (e.g. 1=1min, 2=5min)', +`counter` UInt32 COMMENT 'Number of consecutive intervals this alert condition has been detected', +`description` String COMMENT 'Human-readable description of the alert', +`json` String COMMENT 'Additional alert context and metadata as a JSON blob', +`user_label` String COMMENT 'User-defined free-text label applied to this alert', +`user_label_tstamp` DateTime COMMENT 'Timestamp when user_label was last set', +`alert_category` UInt8 COMMENT 'Alert category (maps to ntopng AlertCategory enum)', +`require_attention` Boolean COMMENT 'True if this alert has been flagged as requiring manual attention' +) ENGINE = Memory +COMMENT 'In-memory table holding currently active (engaged) user alerts. Rows are inserted when an alert fires and removed when resolved. Merged with user_alerts in user_alerts_view.'; @ CREATE TABLE IF NOT EXISTS `system_alerts` ( -`rowid` UUID, -`alert_id` UInt32, -`alert_status` UInt8, -`interface_id` UInt16 DEFAULT 65535, -`name` String, -`tstamp` DateTime, -`tstamp_end` DateTime, -`severity` UInt8, -`score` UInt16, -`granularity` UInt8, -`counter` UInt32, -`description` String, -`json` String, -`user_label` String, -`user_label_tstamp` DateTime, -`alert_category` UInt8, -`require_attention` Boolean -) ENGINE = MergeTree() PARTITION BY toYYYYMMDD(tstamp) ORDER BY (tstamp); +`rowid` UUID COMMENT 'Unique identifier for this alert row (UUID v4)', +`alert_id` UInt32 COMMENT 'Alert type identifier (maps to ntopng alert type enum)', +`alert_status` UInt8 COMMENT 'Alert lifecycle status (0=engaged/active, 1=released/archived)', +`interface_id` UInt16 DEFAULT 65535 COMMENT 'ntopng interface identifier; 65535 means system/global scope', +`name` String COMMENT 'Name of the subsystem or component that generated the alert', +`tstamp` DateTime COMMENT 'Timestamp when the alert was first triggered (alert start time)', +`tstamp_end` DateTime COMMENT 'Timestamp when the alert was resolved (zero/epoch if still active)', +`severity` UInt8 COMMENT 'Alert severity level (maps to ntopng AlertLevel enum)', +`score` UInt16 COMMENT 'Numeric risk/impact score associated with this alert', +`granularity` UInt8 COMMENT 'Periodic check interval that triggered the alert (e.g. 1=1min, 2=5min)', +`counter` UInt32 COMMENT 'Number of consecutive intervals this alert condition has been detected', +`description` String COMMENT 'Human-readable description of the alert', +`json` String COMMENT 'Additional alert context and metadata as a JSON blob', +`user_label` String COMMENT 'User-defined free-text label applied to this alert', +`user_label_tstamp` DateTime COMMENT 'Timestamp when user_label was last set', +`alert_category` UInt8 COMMENT 'Alert category (maps to ntopng AlertCategory enum)', +`require_attention` Boolean COMMENT 'True if this alert has been flagged as requiring manual attention' +) ENGINE = MergeTree() PARTITION BY toYYYYMMDD(tstamp) ORDER BY (tstamp) +COMMENT 'Historical system-level alerts (e.g. license issues, connectivity failures, internal subsystem events). See engaged_system_alerts for currently-firing alerts and system_alerts_view to query both together.'; @ ALTER TABLE `system_alerts` ADD COLUMN IF NOT EXISTS alert_category UInt8; @ @@ -688,24 +707,25 @@ ALTER TABLE `system_alerts` ADD COLUMN IF NOT EXISTS require_attention Boolean; DROP TABLE IF EXISTS `engaged_system_alerts`; @ CREATE TABLE `engaged_system_alerts` ( -`rowid` UUID, -`alert_id` UInt32, -`alert_status` UInt8, -`interface_id` UInt16 DEFAULT 65535, -`name` String, -`tstamp` DateTime, -`tstamp_end` DateTime, -`severity` UInt8, -`score` UInt16, -`granularity` UInt8, -`counter` UInt32, -`description` String, -`json` String, -`user_label` String, -`user_label_tstamp` DateTime, -`alert_category` UInt8, -`require_attention` Boolean -) ENGINE = Memory; +`rowid` UUID COMMENT 'Unique identifier for this alert row (UUID v4)', +`alert_id` UInt32 COMMENT 'Alert type identifier (maps to ntopng alert type enum)', +`alert_status` UInt8 COMMENT 'Alert lifecycle status (0=engaged/active, 1=released/archived)', +`interface_id` UInt16 DEFAULT 65535 COMMENT 'ntopng interface identifier; 65535 means system/global scope', +`name` String COMMENT 'Name of the subsystem or component that generated the alert', +`tstamp` DateTime COMMENT 'Timestamp when the alert was first triggered (alert start time)', +`tstamp_end` DateTime COMMENT 'Timestamp when the alert was resolved (zero/epoch if still active)', +`severity` UInt8 COMMENT 'Alert severity level (maps to ntopng AlertLevel enum)', +`score` UInt16 COMMENT 'Numeric risk/impact score associated with this alert', +`granularity` UInt8 COMMENT 'Periodic check interval that triggered the alert (e.g. 1=1min, 2=5min)', +`counter` UInt32 COMMENT 'Number of consecutive intervals this alert condition has been detected', +`description` String COMMENT 'Human-readable description of the alert', +`json` String COMMENT 'Additional alert context and metadata as a JSON blob', +`user_label` String COMMENT 'User-defined free-text label applied to this alert', +`user_label_tstamp` DateTime COMMENT 'Timestamp when user_label was last set', +`alert_category` UInt8 COMMENT 'Alert category (maps to ntopng AlertCategory enum)', +`require_attention` Boolean COMMENT 'True if this alert has been flagged as requiring manual attention' +) ENGINE = Memory +COMMENT 'In-memory table holding currently active (engaged) system alerts. Rows are inserted when an alert fires and removed when resolved. Merged with system_alerts in system_alerts_view.'; @ @@ -713,47 +733,48 @@ CREATE TABLE `engaged_system_alerts` ( DROP TABLE IF EXISTS `aggregated_flows`; @ CREATE TABLE IF NOT EXISTS `hourly_flows` ( -`FLOW_ID` UInt64, -`IP_PROTOCOL_VERSION` UInt8, -`FIRST_SEEN` DateTime, -`LAST_SEEN` DateTime, -`VLAN_ID` UInt16, -`PACKETS` UInt32, -`TOTAL_BYTES` UInt64, -`SRC2DST_BYTES` UInt64, /* Total */ -`DST2SRC_BYTES` UInt64, /* Total */ -`SCORE` UInt16, /* Total score */ -`PROTOCOL` UInt8, -`IPV4_SRC_ADDR` UInt32, -`IPV6_SRC_ADDR` IPv6, -`IPV4_DST_ADDR` UInt32, -`IPV6_DST_ADDR` IPv6, -`IP_DST_PORT` UInt16, -`L7_PROTO` UInt16, -`L7_PROTO_MASTER` UInt16, -`NUM_FLOWS` UInt32, /* Total number of flows that have been aggregated */ -`FLOW_RISK` UInt64, /* OS of flow risk */ -`SRC_MAC` UInt64, -`DST_MAC` UInt64, -`PROBE_IP` UInt32, /* EXPORTER_IPV4_ADDRESS */ -`EXPORTER_SITE` UInt16, -`NTOPNG_INSTANCE_NAME` String, -`SRC_COUNTRY_CODE` UInt16, -`DST_COUNTRY_CODE` UInt16, -`SRC_ASN` UInt32, -`DST_ASN` UInt32, -`INPUT_SNMP` UInt32, -`OUTPUT_SNMP` UInt32, -`SRC_NETWORK_ID` UInt32, -`DST_NETWORK_ID` UInt32, -`SRC_LABEL` String, -`DST_LABEL` String, -`INTERFACE_ID` UInt16, -`WLAN_SSID` String, -`WTP_MAC_ADDRESS` UInt64, -`CLIENT_LOCATION` UInt8, -`SERVER_LOCATION` UInt8 -) ENGINE = MergeTree() PARTITION BY toYYYYMMDD(FIRST_SEEN) ORDER BY (FIRST_SEEN, IPV4_SRC_ADDR, IPV4_DST_ADDR); +`FLOW_ID` UInt64 COMMENT 'Unique flow identifier assigned by ntopng', +`IP_PROTOCOL_VERSION` UInt8 COMMENT 'IP version: 4 for IPv4, 6 for IPv6', +`FIRST_SEEN` DateTime COMMENT 'Timestamp of the first packet of the flow', +`LAST_SEEN` DateTime COMMENT 'Timestamp of the last packet of the flow', +`VLAN_ID` UInt16 COMMENT '802.1Q VLAN tag (0 if untagged)', +`PACKETS` UInt32 COMMENT 'Total packet count in both directions', +`TOTAL_BYTES` UInt64 COMMENT 'Total bytes transferred in both directions', +`SRC2DST_BYTES` UInt64 /* Total */ COMMENT 'Bytes sent from client (source) to server (destination)', +`DST2SRC_BYTES` UInt64 /* Total */ COMMENT 'Bytes sent from server (destination) to client (source)', +`SCORE` UInt16 /* Total score */ COMMENT 'Composite flow risk/security score', +`PROTOCOL` UInt8 COMMENT 'IP transport protocol number (6=TCP, 17=UDP, 1=ICMP, etc.)', +`IPV4_SRC_ADDR` UInt32 COMMENT 'Source IPv4 address as a 32-bit integer; 0 for IPv6 flows', +`IPV6_SRC_ADDR` IPv6 COMMENT 'Source IPv6 address; all-zeros for IPv4 flows', +`IPV4_DST_ADDR` UInt32 COMMENT 'Destination IPv4 address as a 32-bit integer; 0 for IPv6 flows', +`IPV6_DST_ADDR` IPv6 COMMENT 'Destination IPv6 address; all-zeros for IPv4 flows', +`IP_DST_PORT` UInt16 COMMENT 'Destination (server) port number', +`L7_PROTO` UInt16 COMMENT 'nDPI layer-7 application protocol identifier', +`L7_PROTO_MASTER` UInt16 COMMENT 'nDPI master/carrier protocol ID (e.g. TLS when L7_PROTO is HTTPS)', +`NUM_FLOWS` UInt32 /* Total number of flows that have been aggregated */ COMMENT 'Number of raw flows aggregated into this hourly summary row', +`FLOW_RISK` UInt64 /* OS of flow risk */ COMMENT 'Bitmap of nDPI flow risk flags (each bit represents a distinct risk)', +`SRC_MAC` UInt64 COMMENT 'Source MAC address encoded as a 64-bit integer', +`DST_MAC` UInt64 COMMENT 'Destination MAC address encoded as a 64-bit integer', +`PROBE_IP` UInt32 /* EXPORTER_IPV4_ADDRESS */ COMMENT 'IPv4 address of the NetFlow/IPFIX exporter (probe); same as EXPORTER_IPV4_ADDRESS', +`EXPORTER_SITE` UInt16 COMMENT 'Site/location identifier of the flow exporter', +`NTOPNG_INSTANCE_NAME` String COMMENT 'Hostname/name of the ntopng instance that captured this flow', +`SRC_COUNTRY_CODE` UInt16 COMMENT 'Source IP geo-country: two ASCII letters packed into a UInt16 (high byte = first letter)', +`DST_COUNTRY_CODE` UInt16 COMMENT 'Destination IP geo-country: two ASCII letters packed into a UInt16 (high byte = first letter)', +`SRC_ASN` UInt32 COMMENT 'Autonomous System Number of the source IP', +`DST_ASN` UInt32 COMMENT 'Autonomous System Number of the destination IP', +`INPUT_SNMP` UInt32 COMMENT 'SNMP input interface index exported via NetFlow/IPFIX', +`OUTPUT_SNMP` UInt32 COMMENT 'SNMP output interface index exported via NetFlow/IPFIX', +`SRC_NETWORK_ID` UInt32 COMMENT 'ntopng local-network ID for the source IP (0 if not a known local network)', +`DST_NETWORK_ID` UInt32 COMMENT 'ntopng local-network ID for the destination IP (0 if not a known local network)', +`SRC_LABEL` String COMMENT 'Resolved hostname or user-defined label for the source host', +`DST_LABEL` String COMMENT 'Resolved hostname or user-defined label for the destination host', +`INTERFACE_ID` UInt16 COMMENT 'ntopng internal interface identifier', +`WLAN_SSID` String COMMENT 'Wireless LAN SSID associated with this flow', +`WTP_MAC_ADDRESS` UInt64 COMMENT 'MAC address of the Wireless Termination Point (access point) as a 64-bit integer', +`CLIENT_LOCATION` UInt8 COMMENT 'Client host location type (local LAN, remote, etc.)', +`SERVER_LOCATION` UInt8 COMMENT 'Server host location type (local LAN, remote, etc.)' +) ENGINE = MergeTree() PARTITION BY toYYYYMMDD(FIRST_SEEN) ORDER BY (FIRST_SEEN, IPV4_SRC_ADDR, IPV4_DST_ADDR) +COMMENT 'Hourly aggregated flow summaries. Multiple raw flows sharing the same 5-tuple are collapsed into one row per hour with summed byte/packet counters and OR-ed risk bitmaps. Used for long-term trend analysis and reduced-resolution historical queries.'; @ ALTER TABLE `hourly_flows` ADD COLUMN IF NOT EXISTS SRC_LABEL String; @ @@ -786,61 +807,67 @@ ALTER TABLE `hourly_flows` ADD COLUMN IF NOT EXISTS `EXPORTER_SITE` UInt16; /* VS */ CREATE TABLE IF NOT EXISTS `vulnerability_scan_data` ( -`HOST` String, -`SCAN_TYPE` String, -`LAST_SCAN` DateTime, -`JSON_INFO` String, -`VS_RESULT_FILE` String -) ENGINE = MergeTree() PARTITION BY toYYYYMMDD(LAST_SCAN) ORDER BY (LAST_SCAN, HOST, SCAN_TYPE); +`HOST` String COMMENT 'IP address or hostname of the scanned target', +`SCAN_TYPE` String COMMENT 'Type of vulnerability scan performed (e.g. nmap, openvas)', +`LAST_SCAN` DateTime COMMENT 'Timestamp of when this scan was last performed', +`JSON_INFO` String COMMENT 'Full scan results as a JSON blob', +`VS_RESULT_FILE` String COMMENT 'Path to the raw scan result file on disk' +) ENGINE = MergeTree() PARTITION BY toYYYYMMDD(LAST_SCAN) ORDER BY (LAST_SCAN, HOST, SCAN_TYPE) +COMMENT 'Per-host vulnerability scan results produced by the ntopng Vulnerability Scanner (VS) module. Each row stores the latest scan output for a given host and scan type as a JSON blob.'; @ CREATE TABLE IF NOT EXISTS `vulnerability_scan_report` ( -`REPORT_NAME` String, -`REPORT_DATE` DateTime, -`REPORT_JSON_INFO` String, -`NUM_SCANNED_HOSTS` UInt32, -`NUM_CVES` UInt32, -`NUM_TCP_PORTS` UInt32, -`NUM_UDP_PORTS` UInt32 -) ENGINE = MergeTree() PARTITION BY toYYYYMMDD(REPORT_DATE) ORDER BY (REPORT_DATE); +`REPORT_NAME` String COMMENT 'User-defined name for this vulnerability scan report', +`REPORT_DATE` DateTime COMMENT 'Timestamp when the report was generated', +`REPORT_JSON_INFO` String COMMENT 'Full report metadata and summary as a JSON blob', +`NUM_SCANNED_HOSTS` UInt32 COMMENT 'Number of hosts scanned in this report', +`NUM_CVES` UInt32 COMMENT 'Total number of CVEs identified across all scanned hosts', +`NUM_TCP_PORTS` UInt32 COMMENT 'Total number of open TCP ports found across all scanned hosts', +`NUM_UDP_PORTS` UInt32 COMMENT 'Total number of open UDP ports found across all scanned hosts' +) ENGINE = MergeTree() PARTITION BY toYYYYMMDD(REPORT_DATE) ORDER BY (REPORT_DATE) +COMMENT 'Summary reports of completed vulnerability scans. Each row represents one scan report with aggregate counts of scanned hosts, CVEs found, and open TCP/UDP ports.'; @ /* MITRE */ CREATE TABLE IF NOT EXISTS `mitre_table_info` ( -`ALERT_ID` UInt16, -`ENTITY_ID` UInt16, -`TACTIC` UInt16, -`TECHNIQUE` UInt16, -`SUB_TECHNIQUE` UInt16, -`MITRE_ID` String -) ENGINE = ReplacingMergeTree() PRIMARY KEY (ALERT_ID, ENTITY_ID) ORDER BY (ALERT_ID, ENTITY_ID); +`ALERT_ID` UInt16 COMMENT 'ntopng alert type ID that maps to this MITRE entry', +`ENTITY_ID` UInt16 COMMENT 'ntopng entity type ID (e.g. 1=host, 4=flow) for this mapping', +`TACTIC` UInt16 COMMENT 'MITRE ATT&CK tactic identifier', +`TECHNIQUE` UInt16 COMMENT 'MITRE ATT&CK technique identifier', +`SUB_TECHNIQUE` UInt16 COMMENT 'MITRE ATT&CK sub-technique identifier (0 if none)', +`MITRE_ID` String COMMENT 'MITRE ATT&CK ID string (e.g. T1046, T1595.002)' +) ENGINE = ReplacingMergeTree() PRIMARY KEY (ALERT_ID, ENTITY_ID) ORDER BY (ALERT_ID, ENTITY_ID) +COMMENT 'Mapping of ntopng alert IDs and entity types to MITRE ATT&CK tactics, techniques, and sub-techniques. Joined by host_alerts_view and flow_alerts_view to enrich alert rows with ATT&CK context.'; @ /* ASSET */ CREATE TABLE IF NOT EXISTS `assets` ( -`type` String, -`key` String, -`ifid` UInt8, -`ip` String DEFAULT '', -`mac` String, -`vlan` UInt16 DEFAULT 0, -`network` UInt16 DEFAULT 0, -`name` String DEFAULT '', -`device_type` UInt16 DEFAULT 0, -`manufacturer` String DEFAULT '', -`first_seen` DateTime, -`last_seen` DateTime, -`gateway_mac` String DEFAULT '', -`json_info` String DEFAULT '', -- A json containing all other info -`version` UInt64, -- Used to not have duplicates -`os_type` String DEFAULT '', -`model` String DEFAULT '' -) ENGINE = ReplacingMergeTree(version) PRIMARY KEY (`type`, `key`) ORDER BY (`type`, `key`); +`type` String COMMENT 'Asset category (e.g. host, mac, network_device)', +`key` String COMMENT 'Unique asset key within its type (e.g. IP address, MAC address)', +`ifid` UInt8 COMMENT 'ntopng interface on which this asset was observed', +`ip` String DEFAULT '' COMMENT 'IP address of the asset (empty if not applicable)', +`mac` String COMMENT 'MAC address of the asset', +`vlan` UInt16 DEFAULT 0 COMMENT 'VLAN on which the asset was observed (0 if untagged)', +`network` UInt16 DEFAULT 0 COMMENT 'ntopng local-network ID the asset belongs to', +`name` String DEFAULT '' COMMENT 'Resolved hostname or user-defined name', +`device_type` UInt16 DEFAULT 0 COMMENT 'Device category/type (maps to ntopng DeviceType enum)', +`manufacturer` String DEFAULT '' COMMENT 'Hardware manufacturer derived from MAC OUI lookup', +`first_seen` DateTime COMMENT 'Timestamp when this asset was first observed by ntopng', +`last_seen` DateTime COMMENT 'Timestamp of the most recent observation of this asset', +`gateway_mac` String DEFAULT '' COMMENT 'MAC address of the gateway used to reach this asset', +`json_info` String DEFAULT '' -- A json containing all other info + COMMENT 'Additional asset metadata as a JSON blob (OS info, open ports, etc.)', +`version` UInt64 -- Used to not have duplicates + COMMENT 'Monotonically increasing version counter used by ReplacingMergeTree for deduplication', +`os_type` String DEFAULT '' COMMENT 'Operating system type detected for this asset', +`model` String DEFAULT '' COMMENT 'Hardware model string for this asset' +) ENGINE = ReplacingMergeTree(version) PRIMARY KEY (`type`, `key`) ORDER BY (`type`, `key`) +COMMENT 'Network asset inventory: one row per discovered or imported asset (host, MAC address, network device). Uses ReplacingMergeTree on version so that re-discovered assets update existing rows rather than creating duplicates. json_info holds additional metadata as a JSON blob.'; @ ALTER TABLE assets ADD COLUMN IF NOT EXISTS `os_type` String; @ @@ -1079,24 +1106,25 @@ SELECT 10 entity_id, interface_id, alert_id, alert_status, require_attention, ts /* IMPORTANT: keep in sync with db_schema_as_sqlite.sql */ CREATE TABLE IF NOT EXISTS `hourly_asn` ( -`ID` UInt64, -`NTOPNG_INSTANCE_NAME` String, -`INTERFACE_ID` UInt16, -`IP_PROTOCOL_VERSION` UInt8, -`FIRST_SEEN` DateTime, -`LAST_SEEN` DateTime, -`SRC2DST_BYTES` UInt64, -`DST2SRC_BYTES` UInt64, -`TOTAL_BYTES` UInt64, -`SRC2DST_PACKETS` UInt32, -`DST2SRC_PACKETS` UInt32, -`SRC_ASN` UInt32, -`DST_ASN` UInt32, -`SRC_PEER_ASN` UInt32, -`DST_PEER_ASN` UInt32, -`PROBE_IP` UInt32, /* EXPORTER_IPV4_ADDRESS */ -`INPUT_SNMP` UInt32, -`OUTPUT_SNMP` UInt32 -) ENGINE = MergeTree() PARTITION BY toYYYYMMDD(FIRST_SEEN) ORDER BY (FIRST_SEEN, SRC_ASN, DST_ASN); +`ID` UInt64 COMMENT 'Unique row identifier', +`NTOPNG_INSTANCE_NAME` String COMMENT 'Name of the ntopng instance that generated this record', +`INTERFACE_ID` UInt16 COMMENT 'ntopng interface identifier on which this traffic was observed', +`IP_PROTOCOL_VERSION` UInt8 COMMENT 'IP version: 4 for IPv4, 6 for IPv6', +`FIRST_SEEN` DateTime COMMENT 'Start of the one-hour aggregation window', +`LAST_SEEN` DateTime COMMENT 'End of the one-hour aggregation window', +`SRC2DST_BYTES` UInt64 COMMENT 'Bytes from source ASN to destination ASN in this hour', +`DST2SRC_BYTES` UInt64 COMMENT 'Bytes from destination ASN to source ASN in this hour', +`TOTAL_BYTES` UInt64 COMMENT 'Total bytes between the two ASNs in this hour', +`SRC2DST_PACKETS` UInt32 COMMENT 'Packets from source ASN to destination ASN in this hour', +`DST2SRC_PACKETS` UInt32 COMMENT 'Packets from destination ASN to source ASN in this hour', +`SRC_ASN` UInt32 COMMENT 'Autonomous System Number of the source', +`DST_ASN` UInt32 COMMENT 'Autonomous System Number of the destination', +`SRC_PEER_ASN` UInt32 COMMENT 'BGP peer ASN upstream of the source (EXPORTER_IPV4_ADDRESS)', +`DST_PEER_ASN` UInt32 COMMENT 'BGP peer ASN upstream of the destination', +`PROBE_IP` UInt32 /* EXPORTER_IPV4_ADDRESS */ COMMENT 'IPv4 address of the NetFlow/IPFIX exporter; same as EXPORTER_IPV4_ADDRESS', +`INPUT_SNMP` UInt32 COMMENT 'SNMP input interface index from NetFlow/IPFIX', +`OUTPUT_SNMP` UInt32 COMMENT 'SNMP output interface index from NetFlow/IPFIX' +) ENGINE = MergeTree() PARTITION BY toYYYYMMDD(FIRST_SEEN) ORDER BY (FIRST_SEEN, SRC_ASN, DST_ASN) +COMMENT 'Hourly aggregated traffic statistics per source/destination ASN pair. Used for autonomous-system level traffic analysis and BGP peer analytics. Partitioned by day on FIRST_SEEN.'; @ ALTER TABLE `hourly_asn` ADD COLUMN IF NOT EXISTS TOTAL_BYTES UInt64; diff --git a/httpdocs/misc/db_schema_clickhouse_cluster.sql b/httpdocs/misc/db_schema_clickhouse_cluster.sql index 63de5d40ba..bb98e33b0e 100644 --- a/httpdocs/misc/db_schema_clickhouse_cluster.sql +++ b/httpdocs/misc/db_schema_clickhouse_cluster.sql @@ -1,98 +1,107 @@ +/* Keep flows columns definition in sync with: */ +/* - httpdocs/misc/db_schema_clickhouse.sql */ +/* - pro/include/FlowsTable.h */ +/* - pro/src/ClickHouseDB.cpp -> ClickHouseDB::dumpFlow() */ +/* - scripts/lua/modules/tag_utils.lua -> By adding the new tags for filters */ +/* - scripts/lua/modules/historical_flow_utils.lua -> Columns mapping */ +/* - doc/README.clickhouse_schema.md */ + USE ntopng; @ CREATE TABLE IF NOT EXISTS `flows` ON CLUSTER '$CLUSTER' ( -`FLOW_ID` UInt64, -`IP_PROTOCOL_VERSION` UInt8, -`FIRST_SEEN` DateTime, -`LAST_SEEN` DateTime, -`VLAN_ID` UInt16, /* LowCardinality */ -`PACKETS` UInt32, -`TOTAL_BYTES` UInt64, -`SRC2DST_BYTES` UInt64, -`DST2SRC_BYTES` UInt64, -`SRC2DST_DSCP` UInt8, -`DST2SRC_DSCP` UInt8, -`PROTOCOL` UInt8, -`IPV4_SRC_ADDR` UInt32, -`IPV6_SRC_ADDR` IPv6, -`IP_SRC_PORT` UInt16, -`IPV4_DST_ADDR` UInt32, -`IPV6_DST_ADDR` IPv6, -`IP_DST_PORT` UInt16, -`L7_PROTO` UInt16, -`L7_PROTO_MASTER` UInt16, -`L7_CATEGORY` UInt16, -`FLOW_RISK` UInt64, -`INFO` String, -`PROFILE` String, -`NTOPNG_INSTANCE_NAME` String, -`INTERFACE_ID` UInt16, -`STATUS` UInt8, -`SRC_COUNTRY_CODE` UInt16, -`DST_COUNTRY_CODE` UInt16, -`SRC_LABEL` String, -`DST_LABEL` String, -`SRC_MAC` UInt64, -`DST_MAC` UInt64, -`COMMUNITY_ID` String, -`SRC_ASN` UInt32, -`DST_ASN` UInt32, -`PROBE_IP` UInt32, /* EXPORTER_IPV4_ADDRESS */ -`EXPORTER_SITE` UInt16, -`INTERFACE_ROLE` UInt8, -`OBSERVATION_POINT_ID` UInt16, -`SRC2DST_TCP_FLAGS` UInt8, -`DST2SRC_TCP_FLAGS` UInt8, -`SCORE` UInt16, -`QOE_SCORE` UInt8, -`CLIENT_NW_LATENCY_US` UInt32, -`SERVER_NW_LATENCY_US` UInt32, -`CLIENT_LOCATION` UInt8, -`SERVER_LOCATION` UInt8, -`SRC_NETWORK_ID` UInt32, -`DST_NETWORK_ID` UInt32, -`CLIENT_FINGERPRINT` String, -`TCP_FINGERPRINT` String, -`INPUT_SNMP` UInt32, -`OUTPUT_SNMP` UInt32, -`SRC_HOST_POOL_ID` UInt16, -`DST_HOST_POOL_ID` UInt16, -`SRC_PROC_NAME` String, -`DST_PROC_NAME` String, -`SRC_PROC_USER_NAME` String, -`DST_PROC_USER_NAME` String, -`ALERTS_MAP` String, -`SEVERITY` UInt8, -`IS_CLI_ATTACKER` UInt8, -`IS_CLI_VICTIM` UInt8, -`IS_CLI_BLACKLISTED` UInt8, -`IS_SRV_ATTACKER` UInt8, -`IS_SRV_VICTIM` UInt8, -`IS_SRV_BLACKLISTED` UInt8, -`ALERT_STATUS` UInt8, -`USER_LABEL` String, -`USER_LABEL_TSTAMP` DateTime, -`PROTOCOL_INFO_JSON` String, -`ALERT_JSON` String, -`IS_ALERT_DELETED` UInt8, -`SRC2DST_PACKETS` UInt32, -`DST2SRC_PACKETS` UInt32, -`ALERT_CATEGORY` UInt8, -`MINOR_CONNECTION_STATE` UInt8, -`MAJOR_CONNECTION_STATE` UInt8, -`POST_NAT_IPV4_SRC_ADDR` UInt32, -`POST_NAT_SRC_PORT` UInt32, -`POST_NAT_IPV4_DST_ADDR` UInt32, -`POST_NAT_DST_PORT` UInt32, -`WLAN_SSID` String, -`WTP_MAC_ADDRESS` UInt64, -`DOMAIN_NAME` String, -`SRC_PEER_ASN` UInt32, -`DST_PEER_ASN` UInt32, -`REQUIRE_ATTENTION` Boolean -) ENGINE = ReplicatedMergeTree('/clickhouse/{cluster}/tables/{database}/{table}', '{replica}') PARTITION BY toYYYYMMDD(FIRST_SEEN) ORDER BY (FIRST_SEEN, IPV4_SRC_ADDR, IPV4_DST_ADDR); +`FLOW_ID` UInt64 COMMENT 'Unique flow identifier assigned by ntopng', +`IP_PROTOCOL_VERSION` UInt8 COMMENT 'IP version: 4 for IPv4, 6 for IPv6', +`FIRST_SEEN` DateTime COMMENT 'Timestamp of the first packet of the flow', +`LAST_SEEN` DateTime COMMENT 'Timestamp of the last packet of the flow', +`VLAN_ID` UInt16 /* LowCardinality */ COMMENT '802.1Q VLAN tag (0 if untagged)', +`PACKETS` UInt32 COMMENT 'Total packet count in both directions', +`TOTAL_BYTES` UInt64 COMMENT 'Total bytes transferred in both directions', +`SRC2DST_BYTES` UInt64 COMMENT 'Bytes sent from client (source) to server (destination)', +`DST2SRC_BYTES` UInt64 COMMENT 'Bytes sent from server (destination) to client (source)', +`SRC2DST_DSCP` UInt8 COMMENT 'DSCP value observed in the client-to-server direction', +`DST2SRC_DSCP` UInt8 COMMENT 'DSCP value observed in the server-to-client direction', +`PROTOCOL` UInt8 COMMENT 'IP transport protocol number (6=TCP, 17=UDP, 1=ICMP, etc.)', +`IPV4_SRC_ADDR` UInt32 COMMENT 'Source IPv4 address as a 32-bit integer; 0 for IPv6 flows', +`IPV6_SRC_ADDR` IPv6 COMMENT 'Source IPv6 address; all-zeros for IPv4 flows', +`IP_SRC_PORT` UInt16 COMMENT 'Source (client) port number', +`IPV4_DST_ADDR` UInt32 COMMENT 'Destination IPv4 address as a 32-bit integer; 0 for IPv6 flows', +`IPV6_DST_ADDR` IPv6 COMMENT 'Destination IPv6 address; all-zeros for IPv4 flows', +`IP_DST_PORT` UInt16 COMMENT 'Destination (server) port number', +`L7_PROTO` UInt16 COMMENT 'nDPI layer-7 application protocol identifier', +`L7_PROTO_MASTER` UInt16 COMMENT 'nDPI master/carrier protocol ID (e.g. TLS when L7_PROTO is HTTPS)', +`L7_CATEGORY` UInt16 COMMENT 'nDPI application category identifier', +`FLOW_RISK` UInt64 COMMENT 'Bitmap of nDPI flow risk flags (each bit represents a distinct risk)', +`INFO` String COMMENT 'Supplementary flow info extracted by nDPI (e.g. HTTP host, DNS query name, TLS SNI)', +`PROFILE` String COMMENT 'Traffic policy profile name matched by this flow', +`NTOPNG_INSTANCE_NAME` String COMMENT 'Hostname/name of the ntopng instance that captured this flow', +`INTERFACE_ID` UInt16 COMMENT 'ntopng internal interface identifier', +`STATUS` UInt8 COMMENT 'Flow alert type ID (0 = normal non-alert flow; non-zero = alert type)', +`SRC_COUNTRY_CODE` UInt16 COMMENT 'Source IP geo-country: two ASCII letters packed into a UInt16 (high byte = first letter)', +`DST_COUNTRY_CODE` UInt16 COMMENT 'Destination IP geo-country: two ASCII letters packed into a UInt16 (high byte = first letter)', +`SRC_LABEL` String COMMENT 'Resolved hostname or user-defined label for the source host', +`DST_LABEL` String COMMENT 'Resolved hostname or user-defined label for the destination host', +`SRC_MAC` UInt64 COMMENT 'Source MAC address encoded as a 64-bit integer', +`DST_MAC` UInt64 COMMENT 'Destination MAC address encoded as a 64-bit integer', +`COMMUNITY_ID` String COMMENT 'Community ID v1 flow hash for cross-tool correlation', +`SRC_ASN` UInt32 COMMENT 'Autonomous System Number of the source IP', +`DST_ASN` UInt32 COMMENT 'Autonomous System Number of the destination IP', +`PROBE_IP` UInt32 /* EXPORTER_IPV4_ADDRESS */ COMMENT 'IPv4 address of the NetFlow/IPFIX exporter (probe); same as EXPORTER_IPV4_ADDRESS', +`EXPORTER_SITE` UInt16 COMMENT 'Site/location identifier of the flow exporter', +`INTERFACE_ROLE` UInt8 COMMENT 'Role of the capturing interface (e.g. LAN, WAN)', +`OBSERVATION_POINT_ID` UInt16 COMMENT 'IPFIX observation point identifier', +`SRC2DST_TCP_FLAGS` UInt8 COMMENT 'Bitwise OR of TCP flags seen in the client-to-server direction', +`DST2SRC_TCP_FLAGS` UInt8 COMMENT 'Bitwise OR of TCP flags seen in the server-to-client direction', +`SCORE` UInt16 COMMENT 'Composite flow risk/security score', +`QOE_SCORE` UInt8 COMMENT 'Quality of Experience score (0=best)', +`CLIENT_NW_LATENCY_US` UInt32 COMMENT 'Estimated client-side network RTT in microseconds', +`SERVER_NW_LATENCY_US` UInt32 COMMENT 'Estimated server-side network RTT in microseconds', +`CLIENT_LOCATION` UInt8 COMMENT 'Client host location type (local LAN, remote, etc.)', +`SERVER_LOCATION` UInt8 COMMENT 'Server host location type (local LAN, remote, etc.)', +`SRC_NETWORK_ID` UInt32 COMMENT 'ntopng local-network ID for the source IP (0 if not a known local network)', +`DST_NETWORK_ID` UInt32 COMMENT 'ntopng local-network ID for the destination IP (0 if not a known local network)', +`CLIENT_FINGERPRINT` String COMMENT 'TLS/QUIC client fingerprint (JA3 or similar)', +`TCP_FINGERPRINT` String COMMENT 'TCP stack fingerprint used for passive OS detection', +`INPUT_SNMP` UInt32 COMMENT 'SNMP input interface index exported via NetFlow/IPFIX', +`OUTPUT_SNMP` UInt32 COMMENT 'SNMP output interface index exported via NetFlow/IPFIX', +`SRC_HOST_POOL_ID` UInt16 COMMENT 'ntopng host-pool ID of the source host', +`DST_HOST_POOL_ID` UInt16 COMMENT 'ntopng host-pool ID of the destination host', +`SRC_PROC_NAME` String COMMENT 'Name of the OS process that originated the flow (from eBPF/sysdig)', +`DST_PROC_NAME` String COMMENT 'Name of the OS process that received the flow (from eBPF/sysdig)', +`SRC_PROC_USER_NAME` String COMMENT 'OS username owning the source process', +`DST_PROC_USER_NAME` String COMMENT 'OS username owning the destination process', +`ALERTS_MAP` String COMMENT 'Serialized bitmap of individual alert conditions triggered on this flow', +`SEVERITY` UInt8 COMMENT 'Alert severity level; meaningful only when STATUS != 0', +`IS_CLI_ATTACKER` UInt8 COMMENT '1 if the client host is flagged as an attacker, 0 otherwise', +`IS_CLI_VICTIM` UInt8 COMMENT '1 if the client host is flagged as a victim, 0 otherwise', +`IS_CLI_BLACKLISTED` UInt8 COMMENT '1 if the client IP appears on a threat-intel blacklist, 0 otherwise', +`IS_SRV_ATTACKER` UInt8 COMMENT '1 if the server host is flagged as an attacker, 0 otherwise', +`IS_SRV_VICTIM` UInt8 COMMENT '1 if the server host is flagged as a victim, 0 otherwise', +`IS_SRV_BLACKLISTED` UInt8 COMMENT '1 if the server IP appears on a threat-intel blacklist, 0 otherwise', +`ALERT_STATUS` UInt8 COMMENT 'Alert lifecycle status (e.g. acknowledged, in-progress)', +`USER_LABEL` String COMMENT 'User-defined free-text label applied to this flow', +`USER_LABEL_TSTAMP` DateTime COMMENT 'Timestamp when USER_LABEL was last modified', +`PROTOCOL_INFO_JSON` String COMMENT 'Protocol-specific metadata (HTTP URL, DNS answers, TLS cert info, etc.) as JSON', +`ALERT_JSON` String COMMENT 'Alert-specific context and evidence as a JSON blob', +`IS_ALERT_DELETED` UInt8 COMMENT '1 if the alert on this flow was manually acknowledged/deleted, 0 otherwise', +`SRC2DST_PACKETS` UInt32 COMMENT 'Packet count from client (source) to server (destination)', +`DST2SRC_PACKETS` UInt32 COMMENT 'Packet count from server (destination) to client (source)', +`ALERT_CATEGORY` UInt8 COMMENT 'Alert category identifier (maps to ntopng AlertCategory enum)', +`MINOR_CONNECTION_STATE` UInt8 COMMENT 'Fine-grained TCP/flow connection state', +`MAJOR_CONNECTION_STATE` UInt8 COMMENT 'Coarse TCP connection state (e.g. established, closing, closed)', +`POST_NAT_IPV4_SRC_ADDR` UInt32 COMMENT 'Source IPv4 address after NAT translation', +`POST_NAT_SRC_PORT` UInt32 COMMENT 'Source port after NAT translation', +`POST_NAT_IPV4_DST_ADDR` UInt32 COMMENT 'Destination IPv4 address after NAT translation', +`POST_NAT_DST_PORT` UInt32 COMMENT 'Destination port after NAT translation', +`WLAN_SSID` String COMMENT 'Wireless LAN SSID associated with this flow', +`WTP_MAC_ADDRESS` UInt64 COMMENT 'MAC address of the Wireless Termination Point (access point) as a 64-bit integer', +`DOMAIN_NAME` String COMMENT 'Domain name extracted from the flow (from SNI, DNS, or HTTP Host header)', +`SRC_PEER_ASN` UInt32 COMMENT 'BGP peer ASN upstream of the source IP', +`DST_PEER_ASN` UInt32 COMMENT 'BGP peer ASN upstream of the destination IP', +`REQUIRE_ATTENTION` Boolean COMMENT 'True if this flow/alert has been flagged as requiring manual review' +) ENGINE = ReplicatedMergeTree('/clickhouse/{cluster}/tables/{database}/{table}', '{replica}') PARTITION BY toYYYYMMDD(FIRST_SEEN) ORDER BY (FIRST_SEEN, IPV4_SRC_ADDR, IPV4_DST_ADDR) +COMMENT 'Per-flow telemetry records captured locally or received via NetFlow/sFlow/IPFIX. Each row represents one bidirectional network flow with 5-tuple (src/dst IP, src/dst port, protocol), byte/packet counters, L7 application identification, flow-risk bitmap, DSCP, NAT addresses, process info, and optional alert metadata. Partitioned by day on FIRST_SEEN.'; @ ALTER TABLE `flows` ON CLUSTER '$CLUSTER' ADD COLUMN IF NOT EXISTS `FLOW_ID` UInt64; @ @@ -203,27 +212,28 @@ ALTER TABLE flows ADD COLUMN IF NOT EXISTS `INTERFACE_ROLE` UInt8; @ CREATE TABLE IF NOT EXISTS `active_monitoring_alerts` ON CLUSTER '$CLUSTER' ( -`rowid` UUID, -`alert_id` UInt32, -`alert_status` UInt8, -`interface_id` UInt16 DEFAULT 65535, -`resolved_ip` String, -`resolved_name` String, -`measurement` String, -`measure_threshold` UInt32 DEFAULT 0, -`measure_value` REAL DEFAULT 0.0, -`tstamp` DateTime, -`tstamp_end` DateTime DEFAULT toDateTime(0), -`severity` UInt8, -`score` UInt16, -`counter` UInt32, -`description` String, -`json` String, -`user_label` String, -`user_label_tstamp` DateTime DEFAULT toDateTime(0), -`alert_category` UInt8, -`require_attention` Boolean -) ENGINE = ReplicatedMergeTree('/clickhouse/{cluster}/tables/{database}/{table}', '{replica}') PARTITION BY toYYYYMMDD(tstamp) ORDER BY (tstamp); +`rowid` UUID COMMENT 'Unique identifier for this alert row (UUID v4)', +`alert_id` UInt32 COMMENT 'Alert type identifier (maps to ntopng alert type enum)', +`alert_status` UInt8 COMMENT 'Alert lifecycle status (0=engaged/active, 1=released/archived)', +`interface_id` UInt16 DEFAULT 65535 COMMENT 'ntopng interface identifier; 65535 means system/global scope', +`resolved_ip` String COMMENT 'IP address resolved from the monitored hostname at time of check', +`resolved_name` String COMMENT 'Hostname or target being monitored (FQDN or IP)', +`measurement` String COMMENT 'Type of active monitoring check (e.g. icmp, http, https, tls)', +`measure_threshold` UInt32 DEFAULT 0 COMMENT 'Configured threshold value that was exceeded to trigger the alert', +`measure_value` REAL DEFAULT 0.0 COMMENT 'Measured value (e.g. latency in ms, HTTP response code) at alert time', +`tstamp` DateTime COMMENT 'Timestamp when the alert was first triggered (alert start time)', +`tstamp_end` DateTime DEFAULT toDateTime(0) COMMENT 'Timestamp when the alert was resolved (zero/epoch if still active)', +`severity` UInt8 COMMENT 'Alert severity level (maps to ntopng AlertLevel enum)', +`score` UInt16 COMMENT 'Numeric risk/impact score associated with this alert', +`counter` UInt32 COMMENT 'Number of consecutive intervals this alert condition has been detected', +`description` String COMMENT 'Human-readable description of the alert', +`json` String COMMENT 'Additional alert context and metadata as a JSON blob', +`user_label` String COMMENT 'User-defined free-text label applied to this alert', +`user_label_tstamp` DateTime DEFAULT toDateTime(0) COMMENT 'Timestamp when user_label was last set', +`alert_category` UInt8 COMMENT 'Alert category (maps to ntopng AlertCategory enum)', +`require_attention` Boolean COMMENT 'True if this alert has been flagged as requiring manual attention' +) ENGINE = ReplicatedMergeTree('/clickhouse/{cluster}/tables/{database}/{table}', '{replica}') PARTITION BY toYYYYMMDD(tstamp) ORDER BY (tstamp) +COMMENT 'Historical alerts generated by the active monitoring subsystem (ICMP ping, HTTP, TLS checks, etc.). Rows are appended when an engaged alert is archived. See engaged_active_monitoring_alerts for currently-firing alerts and active_monitoring_alerts_view to query both together.'; @ ALTER TABLE `active_monitoring_alerts` ON CLUSTER '$CLUSTER' ADD COLUMN IF NOT EXISTS alert_category UInt8; @ @@ -234,59 +244,61 @@ ALTER TABLE `active_monitoring_alerts` ON CLUSTER '$CLUSTER' ADD COLUMN IF NOT E DROP TABLE IF EXISTS `engaged_active_monitoring_alerts`; @ CREATE TABLE `engaged_active_monitoring_alerts` ( -`rowid` UUID, -`alert_id` UInt32, -`alert_status` UInt8, -`interface_id` UInt16 DEFAULT 65535, -`resolved_ip` String, -`resolved_name` String, -`measurement` String, -`measure_threshold` UInt32 DEFAULT 0, -`measure_value` REAL DEFAULT 0.0, -`tstamp` DateTime, -`tstamp_end` DateTime DEFAULT toDateTime(0), -`severity` UInt8, -`score` UInt16, -`counter` UInt32, -`description` String, -`json` String, -`user_label` String, -`user_label_tstamp` DateTime DEFAULT toDateTime(0), -`alert_category` UInt8, -`require_attention` Boolean -) ENGINE = Memory; +`rowid` UUID COMMENT 'Unique identifier for this alert row (UUID v4)', +`alert_id` UInt32 COMMENT 'Alert type identifier (maps to ntopng alert type enum)', +`alert_status` UInt8 COMMENT 'Alert lifecycle status (0=engaged/active, 1=released/archived)', +`interface_id` UInt16 DEFAULT 65535 COMMENT 'ntopng interface identifier; 65535 means system/global scope', +`resolved_ip` String COMMENT 'IP address resolved from the monitored hostname at time of check', +`resolved_name` String COMMENT 'Hostname or target being monitored (FQDN or IP)', +`measurement` String COMMENT 'Type of active monitoring check (e.g. icmp, http, https, tls)', +`measure_threshold` UInt32 DEFAULT 0 COMMENT 'Configured threshold value that was exceeded to trigger the alert', +`measure_value` REAL DEFAULT 0.0 COMMENT 'Measured value (e.g. latency in ms, HTTP response code) at alert time', +`tstamp` DateTime COMMENT 'Timestamp when the alert was first triggered (alert start time)', +`tstamp_end` DateTime DEFAULT toDateTime(0) COMMENT 'Timestamp when the alert was resolved (zero/epoch if still active)', +`severity` UInt8 COMMENT 'Alert severity level (maps to ntopng AlertLevel enum)', +`score` UInt16 COMMENT 'Numeric risk/impact score associated with this alert', +`counter` UInt32 COMMENT 'Number of consecutive intervals this alert condition has been detected', +`description` String COMMENT 'Human-readable description of the alert', +`json` String COMMENT 'Additional alert context and metadata as a JSON blob', +`user_label` String COMMENT 'User-defined free-text label applied to this alert', +`user_label_tstamp` DateTime DEFAULT toDateTime(0) COMMENT 'Timestamp when user_label was last set', +`alert_category` UInt8 COMMENT 'Alert category (maps to ntopng AlertCategory enum)', +`require_attention` Boolean COMMENT 'True if this alert has been flagged as requiring manual attention' +) ENGINE = Memory +COMMENT 'In-memory table holding currently active (engaged) active-monitoring alerts. Rows are inserted when an alert fires and removed when resolved. Merged with active_monitoring_alerts in active_monitoring_alerts_view.'; @ CREATE TABLE IF NOT EXISTS `host_alerts` ON CLUSTER '$CLUSTER' ( -`rowid` UUID, -`alert_id` UInt32, -`alert_status` UInt8, -`interface_id` UInt16 DEFAULT 65535, -`ip_version` UInt8, -`ip` String, -`vlan_id` UInt16, -`name` String, -`is_attacker` UInt8, -`is_victim` UInt8, -`is_client` UInt8, -`is_server` UInt8, -`tstamp` DateTime, -`tstamp_end` DateTime, -`severity` UInt8, -`score` UInt16, -`granularity` UInt8, -`counter` UInt32, -`description` String, -`json` String, -`user_label` String, -`user_label_tstamp` DateTime, -`host_pool_id` UInt16, -`network` UInt16, -`country` String, -`alert_category` UInt8, -`require_attention` Boolean -) ENGINE = ReplicatedMergeTree('/clickhouse/{cluster}/tables/{database}/{table}', '{replica}') PARTITION BY toYYYYMMDD(tstamp) ORDER BY (tstamp); +`rowid` UUID COMMENT 'Unique identifier for this alert row (UUID v4)', +`alert_id` UInt32 COMMENT 'Alert type identifier (maps to ntopng alert type enum)', +`alert_status` UInt8 COMMENT 'Alert lifecycle status (0=engaged/active, 1=released/archived)', +`interface_id` UInt16 DEFAULT 65535 COMMENT 'ntopng interface identifier; 65535 means system/global scope', +`ip_version` UInt8 COMMENT 'IP version of the host: 4 for IPv4, 6 for IPv6', +`ip` String COMMENT 'IP address of the host that triggered the alert', +`vlan_id` UInt16 COMMENT 'VLAN on which the host was observed (0 if untagged)', +`name` String COMMENT 'Resolved hostname or user-defined name for the host', +`is_attacker` UInt8 COMMENT '1 if the host is the attacking party in this alert', +`is_victim` UInt8 COMMENT '1 if the host is the victim party in this alert', +`is_client` UInt8 COMMENT '1 if the host acted as a client in the triggering flow', +`is_server` UInt8 COMMENT '1 if the host acted as a server in the triggering flow', +`tstamp` DateTime COMMENT 'Timestamp when the alert was first triggered (alert start time)', +`tstamp_end` DateTime COMMENT 'Timestamp when the alert was resolved (zero/epoch if still active)', +`severity` UInt8 COMMENT 'Alert severity level (maps to ntopng AlertLevel enum)', +`score` UInt16 COMMENT 'Numeric risk/impact score associated with this alert', +`granularity` UInt8 COMMENT 'Periodic check interval that triggered the alert (e.g. 1=1min, 2=5min)', +`counter` UInt32 COMMENT 'Number of consecutive intervals this alert condition has been detected', +`description` String COMMENT 'Human-readable description of the alert', +`json` String COMMENT 'Additional alert context and metadata as a JSON blob', +`user_label` String COMMENT 'User-defined free-text label applied to this alert', +`user_label_tstamp` DateTime COMMENT 'Timestamp when user_label was last set', +`host_pool_id` UInt16 COMMENT 'ntopng host-pool ID the host belongs to', +`network` UInt16 COMMENT 'ntopng local-network ID the host belongs to', +`country` String COMMENT 'Two-letter ISO 3166-1 country code derived from the host IP', +`alert_category` UInt8 COMMENT 'Alert category (maps to ntopng AlertCategory enum)', +`require_attention` Boolean COMMENT 'True if this alert has been flagged as requiring manual attention' +) ENGINE = ReplicatedMergeTree('/clickhouse/{cluster}/tables/{database}/{table}', '{replica}') PARTITION BY toYYYYMMDD(tstamp) ORDER BY (tstamp) +COMMENT 'Historical alerts associated with individual hosts (identified by IP address and VLAN). Rows are appended when an engaged host alert is archived. See engaged_host_alerts for currently-firing alerts and host_alerts_view to query both together (with MITRE ATT&CK enrichment).'; @ ALTER TABLE `host_alerts` ON CLUSTER '$CLUSTER' ADD COLUMN IF NOT EXISTS host_pool_id UInt16; @ @@ -305,60 +317,62 @@ DROP TABLE IF EXISTS `engaged_host_alerts`; @ CREATE TABLE `engaged_host_alerts` ( -`rowid` UUID, -`alert_id` UInt32, -`alert_status` UInt8, -`interface_id` UInt16 DEFAULT 65535, -`ip_version` UInt8, -`ip` String, -`vlan_id` UInt16, -`name` String, -`is_attacker` UInt8, -`is_victim` UInt8, -`is_client` UInt8, -`is_server` UInt8, -`tstamp` DateTime, -`tstamp_end` DateTime, -`severity` UInt8, -`score` UInt16, -`granularity` UInt8, -`counter` UInt32, -`description` String, -`json` String, -`user_label` String, -`user_label_tstamp` DateTime, -`host_pool_id` UInt16, -`network` UInt16, -`country` String, -`alert_category` UInt8, -`require_attention` Boolean -) ENGINE = Memory; +`rowid` UUID COMMENT 'Unique identifier for this alert row (UUID v4)', +`alert_id` UInt32 COMMENT 'Alert type identifier (maps to ntopng alert type enum)', +`alert_status` UInt8 COMMENT 'Alert lifecycle status (0=engaged/active, 1=released/archived)', +`interface_id` UInt16 DEFAULT 65535 COMMENT 'ntopng interface identifier; 65535 means system/global scope', +`ip_version` UInt8 COMMENT 'IP version of the host: 4 for IPv4, 6 for IPv6', +`ip` String COMMENT 'IP address of the host that triggered the alert', +`vlan_id` UInt16 COMMENT 'VLAN on which the host was observed (0 if untagged)', +`name` String COMMENT 'Resolved hostname or user-defined name for the host', +`is_attacker` UInt8 COMMENT '1 if the host is the attacking party in this alert', +`is_victim` UInt8 COMMENT '1 if the host is the victim party in this alert', +`is_client` UInt8 COMMENT '1 if the host acted as a client in the triggering flow', +`is_server` UInt8 COMMENT '1 if the host acted as a server in the triggering flow', +`tstamp` DateTime COMMENT 'Timestamp when the alert was first triggered (alert start time)', +`tstamp_end` DateTime COMMENT 'Timestamp when the alert was resolved (zero/epoch if still active)', +`severity` UInt8 COMMENT 'Alert severity level (maps to ntopng AlertLevel enum)', +`score` UInt16 COMMENT 'Numeric risk/impact score associated with this alert', +`granularity` UInt8 COMMENT 'Periodic check interval that triggered the alert (e.g. 1=1min, 2=5min)', +`counter` UInt32 COMMENT 'Number of consecutive intervals this alert condition has been detected', +`description` String COMMENT 'Human-readable description of the alert', +`json` String COMMENT 'Additional alert context and metadata as a JSON blob', +`user_label` String COMMENT 'User-defined free-text label applied to this alert', +`user_label_tstamp` DateTime COMMENT 'Timestamp when user_label was last set', +`host_pool_id` UInt16 COMMENT 'ntopng host-pool ID the host belongs to', +`network` UInt16 COMMENT 'ntopng local-network ID the host belongs to', +`country` String COMMENT 'Two-letter ISO 3166-1 country code derived from the host IP', +`alert_category` UInt8 COMMENT 'Alert category (maps to ntopng AlertCategory enum)', +`require_attention` Boolean COMMENT 'True if this alert has been flagged as requiring manual attention' +) ENGINE = Memory +COMMENT 'In-memory table holding currently active (engaged) host alerts. Rows are inserted when an alert fires and removed when resolved. Merged with host_alerts in host_alerts_view.'; @ CREATE TABLE IF NOT EXISTS `mac_alerts` ON CLUSTER '$CLUSTER' ( -`rowid` UUID, -`alert_id` UInt32, -`alert_status` UInt8, -`interface_id` UInt16 DEFAULT 65535, -`address` String, -`device_type` UInt8 DEFAULT 0, -`name` String, -`is_attacker` UInt8, -`is_victim` UInt8, -`tstamp` DateTime, -`tstamp_end` DateTime, -`severity` UInt8, -`score` UInt16, -`granularity` UInt8, -`counter` UInt32, -`description` String, -`json` String, -`user_label` String, -`user_label_tstamp` DateTime, -`alert_category` UInt8, -`require_attention` Boolean -) ENGINE = ReplicatedMergeTree('/clickhouse/{cluster}/tables/{database}/{table}', '{replica}') PARTITION BY toYYYYMMDD(tstamp) ORDER BY (tstamp); +`rowid` UUID COMMENT 'Unique identifier for this alert row (UUID v4)', +`alert_id` UInt32 COMMENT 'Alert type identifier (maps to ntopng alert type enum)', +`alert_status` UInt8 COMMENT 'Alert lifecycle status (0=engaged/active, 1=released/archived)', +`interface_id` UInt16 DEFAULT 65535 COMMENT 'ntopng interface identifier; 65535 means system/global scope', +`address` String COMMENT 'MAC address of the device that triggered the alert (colon-separated hex)', +`device_type` UInt8 DEFAULT 0 COMMENT 'Device category/type identifier (maps to ntopng DeviceType enum)', +`name` String COMMENT 'User-defined or discovered name for the device', +`is_attacker` UInt8 COMMENT '1 if the device is the attacking party in this alert', +`is_victim` UInt8 COMMENT '1 if the device is the victim party in this alert', +`tstamp` DateTime COMMENT 'Timestamp when the alert was first triggered (alert start time)', +`tstamp_end` DateTime COMMENT 'Timestamp when the alert was resolved (zero/epoch if still active)', +`severity` UInt8 COMMENT 'Alert severity level (maps to ntopng AlertLevel enum)', +`score` UInt16 COMMENT 'Numeric risk/impact score associated with this alert', +`granularity` UInt8 COMMENT 'Periodic check interval that triggered the alert (e.g. 1=1min, 2=5min)', +`counter` UInt32 COMMENT 'Number of consecutive intervals this alert condition has been detected', +`description` String COMMENT 'Human-readable description of the alert', +`json` String COMMENT 'Additional alert context and metadata as a JSON blob', +`user_label` String COMMENT 'User-defined free-text label applied to this alert', +`user_label_tstamp` DateTime COMMENT 'Timestamp when user_label was last set', +`alert_category` UInt8 COMMENT 'Alert category (maps to ntopng AlertCategory enum)', +`require_attention` Boolean COMMENT 'True if this alert has been flagged as requiring manual attention' +) ENGINE = ReplicatedMergeTree('/clickhouse/{cluster}/tables/{database}/{table}', '{replica}') PARTITION BY toYYYYMMDD(tstamp) ORDER BY (tstamp) +COMMENT 'Historical alerts associated with MAC addresses and layer-2 devices. See engaged_mac_alerts for currently-firing alerts and mac_alerts_view to query both together.'; @ ALTER TABLE `mac_alerts` ON CLUSTER '$CLUSTER' ADD COLUMN IF NOT EXISTS alert_category UInt8; @ @@ -369,53 +383,55 @@ ALTER TABLE `mac_alerts` ON CLUSTER '$CLUSTER' ADD COLUMN IF NOT EXISTS require_ DROP TABLE IF EXISTS `engaged_mac_alerts`; @ CREATE TABLE `engaged_mac_alerts` ( -`rowid` UUID, -`alert_id` UInt32, -`alert_status` UInt8, -`interface_id` UInt16 DEFAULT 65535, -`address` String, -`device_type` UInt8 DEFAULT 0, -`name` String, -`is_attacker` UInt8, -`is_victim` UInt8, -`tstamp` DateTime, -`tstamp_end` DateTime, -`severity` UInt8, -`score` UInt16, -`granularity` UInt8, -`counter` UInt32, -`description` String, -`json` String, -`user_label` String, -`user_label_tstamp` DateTime, -`alert_category` UInt8, -`require_attention` Boolean -) ENGINE = Memory; +`rowid` UUID COMMENT 'Unique identifier for this alert row (UUID v4)', +`alert_id` UInt32 COMMENT 'Alert type identifier (maps to ntopng alert type enum)', +`alert_status` UInt8 COMMENT 'Alert lifecycle status (0=engaged/active, 1=released/archived)', +`interface_id` UInt16 DEFAULT 65535 COMMENT 'ntopng interface identifier; 65535 means system/global scope', +`address` String COMMENT 'MAC address of the device that triggered the alert (colon-separated hex)', +`device_type` UInt8 DEFAULT 0 COMMENT 'Device category/type identifier (maps to ntopng DeviceType enum)', +`name` String COMMENT 'User-defined or discovered name for the device', +`is_attacker` UInt8 COMMENT '1 if the device is the attacking party in this alert', +`is_victim` UInt8 COMMENT '1 if the device is the victim party in this alert', +`tstamp` DateTime COMMENT 'Timestamp when the alert was first triggered (alert start time)', +`tstamp_end` DateTime COMMENT 'Timestamp when the alert was resolved (zero/epoch if still active)', +`severity` UInt8 COMMENT 'Alert severity level (maps to ntopng AlertLevel enum)', +`score` UInt16 COMMENT 'Numeric risk/impact score associated with this alert', +`granularity` UInt8 COMMENT 'Periodic check interval that triggered the alert (e.g. 1=1min, 2=5min)', +`counter` UInt32 COMMENT 'Number of consecutive intervals this alert condition has been detected', +`description` String COMMENT 'Human-readable description of the alert', +`json` String COMMENT 'Additional alert context and metadata as a JSON blob', +`user_label` String COMMENT 'User-defined free-text label applied to this alert', +`user_label_tstamp` DateTime COMMENT 'Timestamp when user_label was last set', +`alert_category` UInt8 COMMENT 'Alert category (maps to ntopng AlertCategory enum)', +`require_attention` Boolean COMMENT 'True if this alert has been flagged as requiring manual attention' +) ENGINE = Memory +COMMENT 'In-memory table holding currently active (engaged) MAC/device alerts. Rows are inserted when an alert fires and removed when resolved. Merged with mac_alerts in mac_alerts_view.'; @ CREATE TABLE IF NOT EXISTS `snmp_alerts` ON CLUSTER '$CLUSTER' ( -`rowid` UUID, -`alert_id` UInt32, -`alert_status` UInt8, -`interface_id` UInt16 DEFAULT 65535, -`ip` String, -`port` UInt32, -`name` String, -`port_name` String, -`tstamp` DateTime, -`tstamp_end` DateTime, -`severity` UInt8, -`score` UInt16, -`granularity` UInt8, -`counter` UInt32, -`description` String, -`json` String, -`user_label` String, -`user_label_tstamp` DateTime, -`alert_category` UInt8, -`require_attention` Boolean -) ENGINE = ReplicatedMergeTree('/clickhouse/{cluster}/tables/{database}/{table}', '{replica}') PARTITION BY toYYYYMMDD(tstamp) ORDER BY (tstamp); +`rowid` UUID COMMENT 'Unique identifier for this alert row (UUID v4)', +`alert_id` UInt32 COMMENT 'Alert type identifier (maps to ntopng alert type enum)', +`alert_status` UInt8 COMMENT 'Alert lifecycle status (0=engaged/active, 1=released/archived)', +`interface_id` UInt16 DEFAULT 65535 COMMENT 'ntopng interface identifier; 65535 means system/global scope', +`ip` String COMMENT 'IP address of the SNMP-polled device that triggered the alert', +`port` UInt32 COMMENT 'SNMP interface index (ifIndex) of the interface that triggered the alert', +`name` String COMMENT 'SNMP sysName or user-defined name of the device', +`port_name` String COMMENT 'SNMP ifDescr or user-defined name of the triggering interface', +`tstamp` DateTime COMMENT 'Timestamp when the alert was first triggered (alert start time)', +`tstamp_end` DateTime COMMENT 'Timestamp when the alert was resolved (zero/epoch if still active)', +`severity` UInt8 COMMENT 'Alert severity level (maps to ntopng AlertLevel enum)', +`score` UInt16 COMMENT 'Numeric risk/impact score associated with this alert', +`granularity` UInt8 COMMENT 'Periodic check interval that triggered the alert (e.g. 1=1min, 2=5min)', +`counter` UInt32 COMMENT 'Number of consecutive intervals this alert condition has been detected', +`description` String COMMENT 'Human-readable description of the alert', +`json` String COMMENT 'Additional alert context and metadata as a JSON blob', +`user_label` String COMMENT 'User-defined free-text label applied to this alert', +`user_label_tstamp` DateTime COMMENT 'Timestamp when user_label was last set', +`alert_category` UInt8 COMMENT 'Alert category (maps to ntopng AlertCategory enum)', +`require_attention` Boolean COMMENT 'True if this alert has been flagged as requiring manual attention' +) ENGINE = ReplicatedMergeTree('/clickhouse/{cluster}/tables/{database}/{table}', '{replica}') PARTITION BY toYYYYMMDD(tstamp) ORDER BY (tstamp) +COMMENT 'Historical alerts from SNMP-polled network devices and their individual ports. See engaged_snmp_alerts for currently-firing alerts and snmp_alerts_view to query both together.'; @ ALTER TABLE `snmp_alerts` MODIFY COLUMN `port` UInt32; @ @@ -428,51 +444,53 @@ ALTER TABLE `snmp_alerts` ON CLUSTER '$CLUSTER' ADD COLUMN IF NOT EXISTS require DROP TABLE IF EXISTS `engaged_snmp_alerts`; @ CREATE TABLE `engaged_snmp_alerts` ( -`rowid` UUID, -`alert_id` UInt32, -`alert_status` UInt8, -`interface_id` UInt16 DEFAULT 65535, -`ip` String, -`port` UInt32, -`name` String, -`port_name` String, -`tstamp` DateTime, -`tstamp_end` DateTime, -`severity` UInt8, -`score` UInt16, -`granularity` UInt8, -`counter` UInt32, -`description` String, -`json` String, -`user_label` String, -`user_label_tstamp` DateTime, -`alert_category` UInt8, -`require_attention` Boolean -) ENGINE = Memory; +`rowid` UUID COMMENT 'Unique identifier for this alert row (UUID v4)', +`alert_id` UInt32 COMMENT 'Alert type identifier (maps to ntopng alert type enum)', +`alert_status` UInt8 COMMENT 'Alert lifecycle status (0=engaged/active, 1=released/archived)', +`interface_id` UInt16 DEFAULT 65535 COMMENT 'ntopng interface identifier; 65535 means system/global scope', +`ip` String COMMENT 'IP address of the SNMP-polled device that triggered the alert', +`port` UInt32 COMMENT 'SNMP interface index (ifIndex) of the interface that triggered the alert', +`name` String COMMENT 'SNMP sysName or user-defined name of the device', +`port_name` String COMMENT 'SNMP ifDescr or user-defined name of the triggering interface', +`tstamp` DateTime COMMENT 'Timestamp when the alert was first triggered (alert start time)', +`tstamp_end` DateTime COMMENT 'Timestamp when the alert was resolved (zero/epoch if still active)', +`severity` UInt8 COMMENT 'Alert severity level (maps to ntopng AlertLevel enum)', +`score` UInt16 COMMENT 'Numeric risk/impact score associated with this alert', +`granularity` UInt8 COMMENT 'Periodic check interval that triggered the alert (e.g. 1=1min, 2=5min)', +`counter` UInt32 COMMENT 'Number of consecutive intervals this alert condition has been detected', +`description` String COMMENT 'Human-readable description of the alert', +`json` String COMMENT 'Additional alert context and metadata as a JSON blob', +`user_label` String COMMENT 'User-defined free-text label applied to this alert', +`user_label_tstamp` DateTime COMMENT 'Timestamp when user_label was last set', +`alert_category` UInt8 COMMENT 'Alert category (maps to ntopng AlertCategory enum)', +`require_attention` Boolean COMMENT 'True if this alert has been flagged as requiring manual attention' +) ENGINE = Memory +COMMENT 'In-memory table holding currently active (engaged) SNMP alerts. Rows are inserted when an alert fires and removed when resolved. Merged with snmp_alerts in snmp_alerts_view.'; @ CREATE TABLE IF NOT EXISTS `network_alerts` ON CLUSTER '$CLUSTER' ( -`rowid` UUID, -`local_network_id` UInt16, -`alert_id` UInt32, -`alert_status` UInt8, -`interface_id` UInt16 DEFAULT 65535, -`name` String, -`alias` String, -`tstamp` DateTime, -`tstamp_end` DateTime, -`severity` UInt8, -`score` UInt16, -`granularity` UInt8, -`counter` UInt32, -`description` String, -`json` String, -`user_label` String, -`user_label_tstamp` DateTime, -`alert_category` UInt8, -`require_attention` Boolean -) ENGINE = ReplicatedMergeTree('/clickhouse/{cluster}/tables/{database}/{table}', '{replica}') PARTITION BY toYYYYMMDD(tstamp) ORDER BY (tstamp); +`rowid` UUID COMMENT 'Unique identifier for this alert row (UUID v4)', +`local_network_id` UInt16 COMMENT 'ntopng internal identifier of the local network subnet', +`alert_id` UInt32 COMMENT 'Alert type identifier (maps to ntopng alert type enum)', +`alert_status` UInt8 COMMENT 'Alert lifecycle status (0=engaged/active, 1=released/archived)', +`interface_id` UInt16 DEFAULT 65535 COMMENT 'ntopng interface identifier; 65535 means system/global scope', +`name` String COMMENT 'CIDR notation or user-defined name of the network (e.g. 192.168.1.0/24)', +`alias` String COMMENT 'User-defined alias for the network', +`tstamp` DateTime COMMENT 'Timestamp when the alert was first triggered (alert start time)', +`tstamp_end` DateTime COMMENT 'Timestamp when the alert was resolved (zero/epoch if still active)', +`severity` UInt8 COMMENT 'Alert severity level (maps to ntopng AlertLevel enum)', +`score` UInt16 COMMENT 'Numeric risk/impact score associated with this alert', +`granularity` UInt8 COMMENT 'Periodic check interval that triggered the alert (e.g. 1=1min, 2=5min)', +`counter` UInt32 COMMENT 'Number of consecutive intervals this alert condition has been detected', +`description` String COMMENT 'Human-readable description of the alert', +`json` String COMMENT 'Additional alert context and metadata as a JSON blob', +`user_label` String COMMENT 'User-defined free-text label applied to this alert', +`user_label_tstamp` DateTime COMMENT 'Timestamp when user_label was last set', +`alert_category` UInt8 COMMENT 'Alert category (maps to ntopng AlertCategory enum)', +`require_attention` Boolean COMMENT 'True if this alert has been flagged as requiring manual attention' +) ENGINE = ReplicatedMergeTree('/clickhouse/{cluster}/tables/{database}/{table}', '{replica}') PARTITION BY toYYYYMMDD(tstamp) ORDER BY (tstamp) +COMMENT 'Historical alerts associated with local network subnets (identified by local_network_id). See engaged_network_alerts for currently-firing alerts and network_alerts_view to query both together.'; @ ALTER TABLE `network_alerts` ON CLUSTER '$CLUSTER' ADD COLUMN IF NOT EXISTS alert_category UInt8; @ @@ -483,101 +501,105 @@ ALTER TABLE `network_alerts` ON CLUSTER '$CLUSTER' ADD COLUMN IF NOT EXISTS requ DROP TABLE IF EXISTS `engaged_network_alerts`; @ CREATE TABLE `engaged_network_alerts` ( -`rowid` UUID, -`local_network_id` UInt16, -`alert_id` UInt32, -`alert_status` UInt8, -`interface_id` UInt16 DEFAULT 65535, -`name` String, -`alias` String, -`tstamp` DateTime, -`tstamp_end` DateTime, -`severity` UInt8, -`score` UInt16, -`granularity` UInt8, -`counter` UInt32, -`description` String, -`json` String, -`user_label` String, -`user_label_tstamp` DateTime, -`alert_category` UInt8, -`require_attention` Boolean -) ENGINE = Memory; +`rowid` UUID COMMENT 'Unique identifier for this alert row (UUID v4)', +`local_network_id` UInt16 COMMENT 'ntopng internal identifier of the local network subnet', +`alert_id` UInt32 COMMENT 'Alert type identifier (maps to ntopng alert type enum)', +`alert_status` UInt8 COMMENT 'Alert lifecycle status (0=engaged/active, 1=released/archived)', +`interface_id` UInt16 DEFAULT 65535 COMMENT 'ntopng interface identifier; 65535 means system/global scope', +`name` String COMMENT 'CIDR notation or user-defined name of the network (e.g. 192.168.1.0/24)', +`alias` String COMMENT 'User-defined alias for the network', +`tstamp` DateTime COMMENT 'Timestamp when the alert was first triggered (alert start time)', +`tstamp_end` DateTime COMMENT 'Timestamp when the alert was resolved (zero/epoch if still active)', +`severity` UInt8 COMMENT 'Alert severity level (maps to ntopng AlertLevel enum)', +`score` UInt16 COMMENT 'Numeric risk/impact score associated with this alert', +`granularity` UInt8 COMMENT 'Periodic check interval that triggered the alert (e.g. 1=1min, 2=5min)', +`counter` UInt32 COMMENT 'Number of consecutive intervals this alert condition has been detected', +`description` String COMMENT 'Human-readable description of the alert', +`json` String COMMENT 'Additional alert context and metadata as a JSON blob', +`user_label` String COMMENT 'User-defined free-text label applied to this alert', +`user_label_tstamp` DateTime COMMENT 'Timestamp when user_label was last set', +`alert_category` UInt8 COMMENT 'Alert category (maps to ntopng AlertCategory enum)', +`require_attention` Boolean COMMENT 'True if this alert has been flagged as requiring manual attention' +) ENGINE = Memory +COMMENT 'In-memory table holding currently active (engaged) network/subnet alerts. Rows are inserted when an alert fires and removed when resolved. Merged with network_alerts in network_alerts_view.'; @ CREATE TABLE IF NOT EXISTS `as_alerts` ON CLUSTER '$CLUSTER' ( -`rowid` UUID, -`asn` UInt32, -`alert_id` UInt32, -`alert_status` UInt8, -`interface_id` UInt16 DEFAULT 65535, -`name` String, -`alias` String, -`tstamp` DateTime, -`tstamp_end` DateTime, -`severity` UInt8, -`score` UInt16, -`granularity` UInt8, -`counter` UInt32, -`description` String, -`json` String, -`user_label` String, -`user_label_tstamp` DateTime, -`alert_category` UInt8, -`require_attention` Boolean -) ENGINE = ReplicatedMergeTree('/clickhouse/{cluster}/tables/{database}/{table}', '{replica}') PARTITION BY toYYYYMMDD(tstamp) ORDER BY (tstamp); +`rowid` UUID COMMENT 'Unique identifier for this alert row (UUID v4)', +`asn` UInt32 COMMENT 'Autonomous System Number that triggered the alert', +`alert_id` UInt32 COMMENT 'Alert type identifier (maps to ntopng alert type enum)', +`alert_status` UInt8 COMMENT 'Alert lifecycle status (0=engaged/active, 1=released/archived)', +`interface_id` UInt16 DEFAULT 65535 COMMENT 'ntopng interface identifier; 65535 means system/global scope', +`name` String COMMENT 'AS name/description (from WHOIS or user configuration)', +`alias` String COMMENT 'User-defined alias for this AS', +`tstamp` DateTime COMMENT 'Timestamp when the alert was first triggered (alert start time)', +`tstamp_end` DateTime COMMENT 'Timestamp when the alert was resolved (zero/epoch if still active)', +`severity` UInt8 COMMENT 'Alert severity level (maps to ntopng AlertLevel enum)', +`score` UInt16 COMMENT 'Numeric risk/impact score associated with this alert', +`granularity` UInt8 COMMENT 'Periodic check interval that triggered the alert (e.g. 1=1min, 2=5min)', +`counter` UInt32 COMMENT 'Number of consecutive intervals this alert condition has been detected', +`description` String COMMENT 'Human-readable description of the alert', +`json` String COMMENT 'Additional alert context and metadata as a JSON blob', +`user_label` String COMMENT 'User-defined free-text label applied to this alert', +`user_label_tstamp` DateTime COMMENT 'Timestamp when user_label was last set', +`alert_category` UInt8 COMMENT 'Alert category (maps to ntopng AlertCategory enum)', +`require_attention` Boolean COMMENT 'True if this alert has been flagged as requiring manual attention' +) ENGINE = ReplicatedMergeTree('/clickhouse/{cluster}/tables/{database}/{table}', '{replica}') PARTITION BY toYYYYMMDD(tstamp) ORDER BY (tstamp) +COMMENT 'Historical alerts associated with Autonomous Systems (identified by ASN). See engaged_as_alerts for currently-firing alerts and as_alerts_view to query both together.'; @ DROP TABLE IF EXISTS `engaged_as_alerts`; @ CREATE TABLE `engaged_as_alerts` ( -`rowid` UUID, -`asn` UInt32, -`alert_id` UInt32, -`alert_status` UInt8, -`interface_id` UInt16 DEFAULT 65535, -`name` String, -`alias` String, -`tstamp` DateTime, -`tstamp_end` DateTime, -`severity` UInt8, -`score` UInt16, -`granularity` UInt8, -`counter` UInt32, -`description` String, -`json` String, -`user_label` String, -`user_label_tstamp` DateTime, -`alert_category` UInt8, -`require_attention` Boolean -) ENGINE = Memory; +`rowid` UUID COMMENT 'Unique identifier for this alert row (UUID v4)', +`asn` UInt32 COMMENT 'Autonomous System Number that triggered the alert', +`alert_id` UInt32 COMMENT 'Alert type identifier (maps to ntopng alert type enum)', +`alert_status` UInt8 COMMENT 'Alert lifecycle status (0=engaged/active, 1=released/archived)', +`interface_id` UInt16 DEFAULT 65535 COMMENT 'ntopng interface identifier; 65535 means system/global scope', +`name` String COMMENT 'AS name/description (from WHOIS or user configuration)', +`alias` String COMMENT 'User-defined alias for this AS', +`tstamp` DateTime COMMENT 'Timestamp when the alert was first triggered (alert start time)', +`tstamp_end` DateTime COMMENT 'Timestamp when the alert was resolved (zero/epoch if still active)', +`severity` UInt8 COMMENT 'Alert severity level (maps to ntopng AlertLevel enum)', +`score` UInt16 COMMENT 'Numeric risk/impact score associated with this alert', +`granularity` UInt8 COMMENT 'Periodic check interval that triggered the alert (e.g. 1=1min, 2=5min)', +`counter` UInt32 COMMENT 'Number of consecutive intervals this alert condition has been detected', +`description` String COMMENT 'Human-readable description of the alert', +`json` String COMMENT 'Additional alert context and metadata as a JSON blob', +`user_label` String COMMENT 'User-defined free-text label applied to this alert', +`user_label_tstamp` DateTime COMMENT 'Timestamp when user_label was last set', +`alert_category` UInt8 COMMENT 'Alert category (maps to ntopng AlertCategory enum)', +`require_attention` Boolean COMMENT 'True if this alert has been flagged as requiring manual attention' +) ENGINE = Memory +COMMENT 'In-memory table holding currently active (engaged) Autonomous System alerts. Rows are inserted when an alert fires and removed when resolved. Merged with as_alerts in as_alerts_view.'; @ CREATE TABLE IF NOT EXISTS `interface_alerts` ON CLUSTER '$CLUSTER' ( -`rowid` UUID, -`ifid` UInt8, -`alert_id` UInt32, -`alert_status` UInt8, -`interface_id` UInt16 DEFAULT 65535, -`subtype` String, -`name` String, -`alias` String, -`tstamp` DateTime, -`tstamp_end` DateTime, -`severity` UInt8, -`score` UInt16, -`granularity` UInt8, -`counter` UInt32, -`description` String, -`json` String, -`user_label` String, -`user_label_tstamp` DateTime, -`alert_category` UInt8, -`require_attention` Boolean -) ENGINE = ReplicatedMergeTree('/clickhouse/{cluster}/tables/{database}/{table}', '{replica}') PARTITION BY toYYYYMMDD(tstamp) ORDER BY (tstamp); +`rowid` UUID COMMENT 'Unique identifier for this alert row (UUID v4)', +`ifid` UInt8 COMMENT 'ntopng internal interface index', +`alert_id` UInt32 COMMENT 'Alert type identifier (maps to ntopng alert type enum)', +`alert_status` UInt8 COMMENT 'Alert lifecycle status (0=engaged/active, 1=released/archived)', +`interface_id` UInt16 DEFAULT 65535 COMMENT 'ntopng interface identifier; 65535 means system/global scope', +`subtype` String COMMENT 'Alert sub-type string providing additional context', +`name` String COMMENT 'Interface name (e.g. eth0, wlan0)', +`alias` String COMMENT 'User-defined alias for the interface', +`tstamp` DateTime COMMENT 'Timestamp when the alert was first triggered (alert start time)', +`tstamp_end` DateTime COMMENT 'Timestamp when the alert was resolved (zero/epoch if still active)', +`severity` UInt8 COMMENT 'Alert severity level (maps to ntopng AlertLevel enum)', +`score` UInt16 COMMENT 'Numeric risk/impact score associated with this alert', +`granularity` UInt8 COMMENT 'Periodic check interval that triggered the alert (e.g. 1=1min, 2=5min)', +`counter` UInt32 COMMENT 'Number of consecutive intervals this alert condition has been detected', +`description` String COMMENT 'Human-readable description of the alert', +`json` String COMMENT 'Additional alert context and metadata as a JSON blob', +`user_label` String COMMENT 'User-defined free-text label applied to this alert', +`user_label_tstamp` DateTime COMMENT 'Timestamp when user_label was last set', +`alert_category` UInt8 COMMENT 'Alert category (maps to ntopng AlertCategory enum)', +`require_attention` Boolean COMMENT 'True if this alert has been flagged as requiring manual attention' +) ENGINE = ReplicatedMergeTree('/clickhouse/{cluster}/tables/{database}/{table}', '{replica}') PARTITION BY toYYYYMMDD(tstamp) ORDER BY (tstamp) +COMMENT 'Historical alerts associated with monitored network interfaces. See engaged_interface_alerts for currently-firing alerts and interface_alerts_view to query both together.'; @ ALTER TABLE `interface_alerts` ON CLUSTER '$CLUSTER' ADD COLUMN IF NOT EXISTS alert_category UInt8; @ @@ -588,49 +610,51 @@ ALTER TABLE `interface_alerts` ON CLUSTER '$CLUSTER' ADD COLUMN IF NOT EXISTS re DROP TABLE IF EXISTS `engaged_interface_alerts`; @ CREATE TABLE `engaged_interface_alerts` ( -`rowid` UUID, -`ifid` UInt8, -`alert_id` UInt32, -`alert_status` UInt8, -`interface_id` UInt16 DEFAULT 65535, -`subtype` String, -`name` String, -`alias` String, -`tstamp` DateTime, -`tstamp_end` DateTime, -`severity` UInt8, -`score` UInt16, -`granularity` UInt8, -`counter` UInt32, -`description` String, -`json` String, -`user_label` String, -`user_label_tstamp` DateTime, -`alert_category` UInt8, -`require_attention` Boolean -) ENGINE = Memory; +`rowid` UUID COMMENT 'Unique identifier for this alert row (UUID v4)', +`ifid` UInt8 COMMENT 'ntopng internal interface index', +`alert_id` UInt32 COMMENT 'Alert type identifier (maps to ntopng alert type enum)', +`alert_status` UInt8 COMMENT 'Alert lifecycle status (0=engaged/active, 1=released/archived)', +`interface_id` UInt16 DEFAULT 65535 COMMENT 'ntopng interface identifier; 65535 means system/global scope', +`subtype` String COMMENT 'Alert sub-type string providing additional context', +`name` String COMMENT 'Interface name (e.g. eth0, wlan0)', +`alias` String COMMENT 'User-defined alias for the interface', +`tstamp` DateTime COMMENT 'Timestamp when the alert was first triggered (alert start time)', +`tstamp_end` DateTime COMMENT 'Timestamp when the alert was resolved (zero/epoch if still active)', +`severity` UInt8 COMMENT 'Alert severity level (maps to ntopng AlertLevel enum)', +`score` UInt16 COMMENT 'Numeric risk/impact score associated with this alert', +`granularity` UInt8 COMMENT 'Periodic check interval that triggered the alert (e.g. 1=1min, 2=5min)', +`counter` UInt32 COMMENT 'Number of consecutive intervals this alert condition has been detected', +`description` String COMMENT 'Human-readable description of the alert', +`json` String COMMENT 'Additional alert context and metadata as a JSON blob', +`user_label` String COMMENT 'User-defined free-text label applied to this alert', +`user_label_tstamp` DateTime COMMENT 'Timestamp when user_label was last set', +`alert_category` UInt8 COMMENT 'Alert category (maps to ntopng AlertCategory enum)', +`require_attention` Boolean COMMENT 'True if this alert has been flagged as requiring manual attention' +) ENGINE = Memory +COMMENT 'In-memory table holding currently active (engaged) interface alerts. Rows are inserted when an alert fires and removed when resolved. Merged with interface_alerts in interface_alerts_view.'; @ CREATE TABLE IF NOT EXISTS `user_alerts` ON CLUSTER '$CLUSTER' ( -`rowid` UUID, -`alert_id` UInt32, -`alert_status` UInt8, -`interface_id` UInt16 DEFAULT 65535, -`user` String, -`tstamp` DateTime, -`tstamp_end` DateTime, -`severity` UInt8, -`score` UInt16, -`granularity` UInt8, -`counter` UInt32, -`description` String, -`json` String, -`user_label` String, -`user_label_tstamp` DateTime, -`alert_category` UInt8, -`require_attention` Boolean -) ENGINE = ReplicatedMergeTree('/clickhouse/{cluster}/tables/{database}/{table}', '{replica}') PARTITION BY toYYYYMMDD(tstamp) ORDER BY (tstamp); +`rowid` UUID COMMENT 'Unique identifier for this alert row (UUID v4)', +`alert_id` UInt32 COMMENT 'Alert type identifier (maps to ntopng alert type enum)', +`alert_status` UInt8 COMMENT 'Alert lifecycle status (0=engaged/active, 1=released/archived)', +`interface_id` UInt16 DEFAULT 65535 COMMENT 'ntopng interface identifier; 65535 means system/global scope', +`user` String COMMENT 'ntopng username associated with this alert', +`tstamp` DateTime COMMENT 'Timestamp when the alert was first triggered (alert start time)', +`tstamp_end` DateTime COMMENT 'Timestamp when the alert was resolved (zero/epoch if still active)', +`severity` UInt8 COMMENT 'Alert severity level (maps to ntopng AlertLevel enum)', +`score` UInt16 COMMENT 'Numeric risk/impact score associated with this alert', +`granularity` UInt8 COMMENT 'Periodic check interval that triggered the alert (e.g. 1=1min, 2=5min)', +`counter` UInt32 COMMENT 'Number of consecutive intervals this alert condition has been detected', +`description` String COMMENT 'Human-readable description of the alert', +`json` String COMMENT 'Additional alert context and metadata as a JSON blob', +`user_label` String COMMENT 'User-defined free-text label applied to this alert', +`user_label_tstamp` DateTime COMMENT 'Timestamp when user_label was last set', +`alert_category` UInt8 COMMENT 'Alert category (maps to ntopng AlertCategory enum)', +`require_attention` Boolean COMMENT 'True if this alert has been flagged as requiring manual attention' +) ENGINE = ReplicatedMergeTree('/clickhouse/{cluster}/tables/{database}/{table}', '{replica}') PARTITION BY toYYYYMMDD(tstamp) ORDER BY (tstamp) +COMMENT 'Historical alerts associated with ntopng-managed users. See engaged_user_alerts for currently-firing alerts and user_alerts_view to query both together.'; @ ALTER TABLE `user_alerts` ON CLUSTER '$CLUSTER' ADD COLUMN IF NOT EXISTS alert_category UInt8; @ @@ -641,46 +665,48 @@ ALTER TABLE `user_alerts` ON CLUSTER '$CLUSTER' ADD COLUMN IF NOT EXISTS require DROP TABLE IF EXISTS `engaged_user_alerts`; @ CREATE TABLE `engaged_user_alerts` ( -`rowid` UUID, -`alert_id` UInt32, -`alert_status` UInt8, -`interface_id` UInt16 DEFAULT 65535, -`user` String, -`tstamp` DateTime, -`tstamp_end` DateTime, -`severity` UInt8, -`score` UInt16, -`granularity` UInt8, -`counter` UInt32, -`description` String, -`json` String, -`user_label` String, -`user_label_tstamp` DateTime, -`alert_category` UInt8, -`require_attention` Boolean -) ENGINE = Memory; +`rowid` UUID COMMENT 'Unique identifier for this alert row (UUID v4)', +`alert_id` UInt32 COMMENT 'Alert type identifier (maps to ntopng alert type enum)', +`alert_status` UInt8 COMMENT 'Alert lifecycle status (0=engaged/active, 1=released/archived)', +`interface_id` UInt16 DEFAULT 65535 COMMENT 'ntopng interface identifier; 65535 means system/global scope', +`user` String COMMENT 'ntopng username associated with this alert', +`tstamp` DateTime COMMENT 'Timestamp when the alert was first triggered (alert start time)', +`tstamp_end` DateTime COMMENT 'Timestamp when the alert was resolved (zero/epoch if still active)', +`severity` UInt8 COMMENT 'Alert severity level (maps to ntopng AlertLevel enum)', +`score` UInt16 COMMENT 'Numeric risk/impact score associated with this alert', +`granularity` UInt8 COMMENT 'Periodic check interval that triggered the alert (e.g. 1=1min, 2=5min)', +`counter` UInt32 COMMENT 'Number of consecutive intervals this alert condition has been detected', +`description` String COMMENT 'Human-readable description of the alert', +`json` String COMMENT 'Additional alert context and metadata as a JSON blob', +`user_label` String COMMENT 'User-defined free-text label applied to this alert', +`user_label_tstamp` DateTime COMMENT 'Timestamp when user_label was last set', +`alert_category` UInt8 COMMENT 'Alert category (maps to ntopng AlertCategory enum)', +`require_attention` Boolean COMMENT 'True if this alert has been flagged as requiring manual attention' +) ENGINE = Memory +COMMENT 'In-memory table holding currently active (engaged) user alerts. Rows are inserted when an alert fires and removed when resolved. Merged with user_alerts in user_alerts_view.'; @ CREATE TABLE IF NOT EXISTS `system_alerts` ON CLUSTER '$CLUSTER' ( -`rowid` UUID, -`alert_id` UInt32, -`alert_status` UInt8, -`interface_id` UInt16 DEFAULT 65535, -`name` String, -`tstamp` DateTime, -`tstamp_end` DateTime, -`severity` UInt8, -`score` UInt16, -`granularity` UInt8, -`counter` UInt32, -`description` String, -`json` String, -`user_label` String, -`user_label_tstamp` DateTime, -`alert_category` UInt8, -`require_attention` Boolean -) ENGINE = ReplicatedMergeTree('/clickhouse/{cluster}/tables/{database}/{table}', '{replica}') PARTITION BY toYYYYMMDD(tstamp) ORDER BY (tstamp); +`rowid` UUID COMMENT 'Unique identifier for this alert row (UUID v4)', +`alert_id` UInt32 COMMENT 'Alert type identifier (maps to ntopng alert type enum)', +`alert_status` UInt8 COMMENT 'Alert lifecycle status (0=engaged/active, 1=released/archived)', +`interface_id` UInt16 DEFAULT 65535 COMMENT 'ntopng interface identifier; 65535 means system/global scope', +`name` String COMMENT 'Name of the subsystem or component that generated the alert', +`tstamp` DateTime COMMENT 'Timestamp when the alert was first triggered (alert start time)', +`tstamp_end` DateTime COMMENT 'Timestamp when the alert was resolved (zero/epoch if still active)', +`severity` UInt8 COMMENT 'Alert severity level (maps to ntopng AlertLevel enum)', +`score` UInt16 COMMENT 'Numeric risk/impact score associated with this alert', +`granularity` UInt8 COMMENT 'Periodic check interval that triggered the alert (e.g. 1=1min, 2=5min)', +`counter` UInt32 COMMENT 'Number of consecutive intervals this alert condition has been detected', +`description` String COMMENT 'Human-readable description of the alert', +`json` String COMMENT 'Additional alert context and metadata as a JSON blob', +`user_label` String COMMENT 'User-defined free-text label applied to this alert', +`user_label_tstamp` DateTime COMMENT 'Timestamp when user_label was last set', +`alert_category` UInt8 COMMENT 'Alert category (maps to ntopng AlertCategory enum)', +`require_attention` Boolean COMMENT 'True if this alert has been flagged as requiring manual attention' +) ENGINE = ReplicatedMergeTree('/clickhouse/{cluster}/tables/{database}/{table}', '{replica}') PARTITION BY toYYYYMMDD(tstamp) ORDER BY (tstamp) +COMMENT 'Historical system-level alerts (e.g. license issues, connectivity failures, internal subsystem events). See engaged_system_alerts for currently-firing alerts and system_alerts_view to query both together.'; @ ALTER TABLE `system_alerts` ON CLUSTER '$CLUSTER' ADD COLUMN IF NOT EXISTS alert_category UInt8; @ @@ -691,71 +717,73 @@ ALTER TABLE `system_alerts` ON CLUSTER '$CLUSTER' ADD COLUMN IF NOT EXISTS requi DROP TABLE IF EXISTS `engaged_system_alerts`; @ CREATE TABLE `engaged_system_alerts` ( -`rowid` UUID, -`alert_id` UInt32, -`alert_status` UInt8, -`interface_id` UInt16 DEFAULT 65535, -`name` String, -`tstamp` DateTime, -`tstamp_end` DateTime, -`severity` UInt8, -`score` UInt16, -`granularity` UInt8, -`counter` UInt32, -`description` String, -`json` String, -`user_label` String, -`user_label_tstamp` DateTime, -`alert_category` UInt8, -`require_attention` Boolean -) ENGINE = Memory; +`rowid` UUID COMMENT 'Unique identifier for this alert row (UUID v4)', +`alert_id` UInt32 COMMENT 'Alert type identifier (maps to ntopng alert type enum)', +`alert_status` UInt8 COMMENT 'Alert lifecycle status (0=engaged/active, 1=released/archived)', +`interface_id` UInt16 DEFAULT 65535 COMMENT 'ntopng interface identifier; 65535 means system/global scope', +`name` String COMMENT 'Name of the subsystem or component that generated the alert', +`tstamp` DateTime COMMENT 'Timestamp when the alert was first triggered (alert start time)', +`tstamp_end` DateTime COMMENT 'Timestamp when the alert was resolved (zero/epoch if still active)', +`severity` UInt8 COMMENT 'Alert severity level (maps to ntopng AlertLevel enum)', +`score` UInt16 COMMENT 'Numeric risk/impact score associated with this alert', +`granularity` UInt8 COMMENT 'Periodic check interval that triggered the alert (e.g. 1=1min, 2=5min)', +`counter` UInt32 COMMENT 'Number of consecutive intervals this alert condition has been detected', +`description` String COMMENT 'Human-readable description of the alert', +`json` String COMMENT 'Additional alert context and metadata as a JSON blob', +`user_label` String COMMENT 'User-defined free-text label applied to this alert', +`user_label_tstamp` DateTime COMMENT 'Timestamp when user_label was last set', +`alert_category` UInt8 COMMENT 'Alert category (maps to ntopng AlertCategory enum)', +`require_attention` Boolean COMMENT 'True if this alert has been flagged as requiring manual attention' +) ENGINE = Memory +COMMENT 'In-memory table holding currently active (engaged) system alerts. Rows are inserted when an alert fires and removed when resolved. Merged with system_alerts in system_alerts_view.'; @ DROP TABLE IF EXISTS `aggregated_flows` ON CLUSTER '$CLUSTER'; @ CREATE TABLE IF NOT EXISTS `hourly_flows` ON CLUSTER '$CLUSTER' ( -`FLOW_ID` UInt64, -`IP_PROTOCOL_VERSION` UInt8, -`FIRST_SEEN` DateTime, -`LAST_SEEN` DateTime, -`VLAN_ID` UInt16, -`PACKETS` UInt32, -`TOTAL_BYTES` UInt64, -`SRC2DST_BYTES` UInt64, /* Total */ -`DST2SRC_BYTES` UInt64, /* Total */ -`SCORE` UInt16, /* Total score */ -`PROTOCOL` UInt8, -`IPV4_SRC_ADDR` UInt32, -`IPV6_SRC_ADDR` IPv6, -`IPV4_DST_ADDR` UInt32, -`IPV6_DST_ADDR` IPv6, -`IP_DST_PORT` UInt16, -`L7_PROTO` UInt16, -`L7_PROTO_MASTER` UInt16, -`NUM_FLOWS` UInt32, /* Total number of flows that have been aggregated */ -`FLOW_RISK` UInt64, /* OS of flow risk */ -`SRC_MAC` UInt64, -`DST_MAC` UInt64, -`PROBE_IP` UInt32, /* EXPORTER_IPV4_ADDRESS */ -`EXPORTER_SITE` UInt16, -`NTOPNG_INSTANCE_NAME` String, -`SRC_COUNTRY_CODE` UInt16, -`DST_COUNTRY_CODE` UInt16, -`SRC_ASN` UInt32, -`DST_ASN` UInt32, -`INPUT_SNMP` UInt32, -`OUTPUT_SNMP` UInt32, -`SRC_NETWORK_ID` UInt32, -`DST_NETWORK_ID` UInt32, -`SRC_LABEL` String, -`DST_LABEL` String, -`INTERFACE_ID` UInt16, -`WLAN_SSID` String, -`WTP_MAC_ADDRESS` UInt64, -`CLIENT_LOCATION` UInt8, -`SERVER_LOCATION` UInt8 -) ENGINE = ReplicatedMergeTree('/clickhouse/{cluster}/tables/{database}/{table}', '{replica}') PARTITION BY toYYYYMMDD(FIRST_SEEN) ORDER BY (FIRST_SEEN, IPV4_SRC_ADDR, IPV4_DST_ADDR); +`FLOW_ID` UInt64 COMMENT 'Unique flow identifier assigned by ntopng', +`IP_PROTOCOL_VERSION` UInt8 COMMENT 'IP version: 4 for IPv4, 6 for IPv6', +`FIRST_SEEN` DateTime COMMENT 'Timestamp of the first packet of the flow', +`LAST_SEEN` DateTime COMMENT 'Timestamp of the last packet of the flow', +`VLAN_ID` UInt16 COMMENT '802.1Q VLAN tag (0 if untagged)', +`PACKETS` UInt32 COMMENT 'Total packet count in both directions', +`TOTAL_BYTES` UInt64 COMMENT 'Total bytes transferred in both directions', +`SRC2DST_BYTES` UInt64 /* Total */ COMMENT 'Bytes sent from client (source) to server (destination)', +`DST2SRC_BYTES` UInt64 /* Total */ COMMENT 'Bytes sent from server (destination) to client (source)', +`SCORE` UInt16 /* Total score */ COMMENT 'Composite flow risk/security score', +`PROTOCOL` UInt8 COMMENT 'IP transport protocol number (6=TCP, 17=UDP, 1=ICMP, etc.)', +`IPV4_SRC_ADDR` UInt32 COMMENT 'Source IPv4 address as a 32-bit integer; 0 for IPv6 flows', +`IPV6_SRC_ADDR` IPv6 COMMENT 'Source IPv6 address; all-zeros for IPv4 flows', +`IPV4_DST_ADDR` UInt32 COMMENT 'Destination IPv4 address as a 32-bit integer; 0 for IPv6 flows', +`IPV6_DST_ADDR` IPv6 COMMENT 'Destination IPv6 address; all-zeros for IPv4 flows', +`IP_DST_PORT` UInt16 COMMENT 'Destination (server) port number', +`L7_PROTO` UInt16 COMMENT 'nDPI layer-7 application protocol identifier', +`L7_PROTO_MASTER` UInt16 COMMENT 'nDPI master/carrier protocol ID (e.g. TLS when L7_PROTO is HTTPS)', +`NUM_FLOWS` UInt32 /* Total number of flows that have been aggregated */ COMMENT 'Number of raw flows aggregated into this hourly summary row', +`FLOW_RISK` UInt64 /* OS of flow risk */ COMMENT 'Bitmap of nDPI flow risk flags (each bit represents a distinct risk)', +`SRC_MAC` UInt64 COMMENT 'Source MAC address encoded as a 64-bit integer', +`DST_MAC` UInt64 COMMENT 'Destination MAC address encoded as a 64-bit integer', +`PROBE_IP` UInt32 /* EXPORTER_IPV4_ADDRESS */ COMMENT 'IPv4 address of the NetFlow/IPFIX exporter (probe); same as EXPORTER_IPV4_ADDRESS', +`EXPORTER_SITE` UInt16 COMMENT 'Site/location identifier of the flow exporter', +`NTOPNG_INSTANCE_NAME` String COMMENT 'Hostname/name of the ntopng instance that captured this flow', +`SRC_COUNTRY_CODE` UInt16 COMMENT 'Source IP geo-country: two ASCII letters packed into a UInt16 (high byte = first letter)', +`DST_COUNTRY_CODE` UInt16 COMMENT 'Destination IP geo-country: two ASCII letters packed into a UInt16 (high byte = first letter)', +`SRC_ASN` UInt32 COMMENT 'Autonomous System Number of the source IP', +`DST_ASN` UInt32 COMMENT 'Autonomous System Number of the destination IP', +`INPUT_SNMP` UInt32 COMMENT 'SNMP input interface index exported via NetFlow/IPFIX', +`OUTPUT_SNMP` UInt32 COMMENT 'SNMP output interface index exported via NetFlow/IPFIX', +`SRC_NETWORK_ID` UInt32 COMMENT 'ntopng local-network ID for the source IP (0 if not a known local network)', +`DST_NETWORK_ID` UInt32 COMMENT 'ntopng local-network ID for the destination IP (0 if not a known local network)', +`SRC_LABEL` String COMMENT 'Resolved hostname or user-defined label for the source host', +`DST_LABEL` String COMMENT 'Resolved hostname or user-defined label for the destination host', +`INTERFACE_ID` UInt16 COMMENT 'ntopng internal interface identifier', +`WLAN_SSID` String COMMENT 'Wireless LAN SSID associated with this flow', +`WTP_MAC_ADDRESS` UInt64 COMMENT 'MAC address of the Wireless Termination Point (access point) as a 64-bit integer', +`CLIENT_LOCATION` UInt8 COMMENT 'Client host location type (local LAN, remote, etc.)', +`SERVER_LOCATION` UInt8 COMMENT 'Server host location type (local LAN, remote, etc.)' +) ENGINE = ReplicatedMergeTree('/clickhouse/{cluster}/tables/{database}/{table}', '{replica}') PARTITION BY toYYYYMMDD(FIRST_SEEN) ORDER BY (FIRST_SEEN, IPV4_SRC_ADDR, IPV4_DST_ADDR) +COMMENT 'Hourly aggregated flow summaries. Multiple raw flows sharing the same 5-tuple are collapsed into one row per hour with summed byte/packet counters and OR-ed risk bitmaps. Used for long-term trend analysis and reduced-resolution historical queries.'; @ ALTER TABLE `hourly_flows` ON CLUSTER '$CLUSTER' ADD COLUMN IF NOT EXISTS SRC_LABEL String; @ @@ -788,61 +816,67 @@ ALTER TABLE `hourly_flows` ON CLUSTER '$CLUSTER' ADD COLUMN IF NOT EXISTS `EXPOR /* VS */ CREATE TABLE IF NOT EXISTS `vulnerability_scan_data` ON CLUSTER '$CLUSTER' ( -`HOST` String, -`SCAN_TYPE` String, -`LAST_SCAN` DateTime, -`JSON_INFO` String, -`VS_RESULT_FILE` String -) ENGINE = ReplicatedMergeTree('/clickhouse/{cluster}/tables/{database}/{table}', '{replica}') PARTITION BY toYYYYMMDD(LAST_SCAN) ORDER BY (LAST_SCAN, HOST, SCAN_TYPE); +`HOST` String COMMENT 'IP address or hostname of the scanned target', +`SCAN_TYPE` String COMMENT 'Type of vulnerability scan performed (e.g. nmap, openvas)', +`LAST_SCAN` DateTime COMMENT 'Timestamp of when this scan was last performed', +`JSON_INFO` String COMMENT 'Full scan results as a JSON blob', +`VS_RESULT_FILE` String COMMENT 'Path to the raw scan result file on disk' +) ENGINE = ReplicatedMergeTree('/clickhouse/{cluster}/tables/{database}/{table}', '{replica}') PARTITION BY toYYYYMMDD(LAST_SCAN) ORDER BY (LAST_SCAN, HOST, SCAN_TYPE) +COMMENT 'Per-host vulnerability scan results produced by the ntopng Vulnerability Scanner (VS) module. Each row stores the latest scan output for a given host and scan type as a JSON blob.'; @ CREATE TABLE IF NOT EXISTS `vulnerability_scan_report` ON CLUSTER '$CLUSTER' ( -`REPORT_NAME` String, -`REPORT_DATE` DateTime, -`REPORT_JSON_INFO` String, -`NUM_SCANNED_HOSTS` UInt32, -`NUM_CVES` UInt32, -`NUM_TCP_PORTS` UInt32, -`NUM_UDP_PORTS` UInt32 -) ENGINE = ReplicatedMergeTree('/clickhouse/{cluster}/tables/{database}/{table}', '{replica}') PARTITION BY toYYYYMMDD(REPORT_DATE) ORDER BY (REPORT_DATE); +`REPORT_NAME` String COMMENT 'User-defined name for this vulnerability scan report', +`REPORT_DATE` DateTime COMMENT 'Timestamp when the report was generated', +`REPORT_JSON_INFO` String COMMENT 'Full report metadata and summary as a JSON blob', +`NUM_SCANNED_HOSTS` UInt32 COMMENT 'Number of hosts scanned in this report', +`NUM_CVES` UInt32 COMMENT 'Total number of CVEs identified across all scanned hosts', +`NUM_TCP_PORTS` UInt32 COMMENT 'Total number of open TCP ports found across all scanned hosts', +`NUM_UDP_PORTS` UInt32 COMMENT 'Total number of open UDP ports found across all scanned hosts' +) ENGINE = ReplicatedMergeTree('/clickhouse/{cluster}/tables/{database}/{table}', '{replica}') PARTITION BY toYYYYMMDD(REPORT_DATE) ORDER BY (REPORT_DATE) +COMMENT 'Summary reports of completed vulnerability scans. Each row represents one scan report with aggregate counts of scanned hosts, CVEs found, and open TCP/UDP ports.'; @ /* MITRE */ CREATE TABLE IF NOT EXISTS `mitre_table_info` ON CLUSTER '$CLUSTER' ( -`ALERT_ID` UInt16, -`ENTITY_ID` UInt16, -`TACTIC` UInt16, -`TECHNIQUE` UInt16, -`SUB_TECHNIQUE` UInt16, -`MITRE_ID` String -) ENGINE = ReplicatedReplacingMergeTree('/clickhouse/{cluster}/tables/{database}/{table}', '{replica}') PRIMARY KEY (`ALERT_ID`, `ENTITY_ID`) ORDER BY (`ALERT_ID`, `ENTITY_ID`); +`ALERT_ID` UInt16 COMMENT 'ntopng alert type ID that maps to this MITRE entry', +`ENTITY_ID` UInt16 COMMENT 'ntopng entity type ID (e.g. 1=host, 4=flow) for this mapping', +`TACTIC` UInt16 COMMENT 'MITRE ATT&CK tactic identifier', +`TECHNIQUE` UInt16 COMMENT 'MITRE ATT&CK technique identifier', +`SUB_TECHNIQUE` UInt16 COMMENT 'MITRE ATT&CK sub-technique identifier (0 if none)', +`MITRE_ID` String COMMENT 'MITRE ATT&CK ID string (e.g. T1046, T1595.002)' +) ENGINE = ReplicatedReplacingMergeTree('/clickhouse/{cluster}/tables/{database}/{table}', '{replica}') PRIMARY KEY (`ALERT_ID`, `ENTITY_ID`) ORDER BY (`ALERT_ID`, `ENTITY_ID`) +COMMENT 'Mapping of ntopng alert IDs and entity types to MITRE ATT&CK tactics, techniques, and sub-techniques. Joined by host_alerts_view and flow_alerts_view to enrich alert rows with ATT&CK context.'; @ /* ASSET */ CREATE TABLE IF NOT EXISTS `assets` ON CLUSTER '$CLUSTER' ( -`type` String, -`key` String, -`ifid` UInt8, -`ip` String DEFAULT '', -`mac` String, -`vlan` UInt16 DEFAULT 0, -`network` UInt16 DEFAULT 0, -`name` String DEFAULT '', -`device_type` UInt16 DEFAULT 0, -`manufacturer` String DEFAULT '', -`first_seen` DateTime, -`last_seen` DateTime, -`gateway_mac` String DEFAULT '', -`json_info` String DEFAULT '', -- A json containing all other info -`version` UInt64, -- Used to not have duplicates -`os_type` String DEFAULT '', -`model` String DEFAULT '' -) ENGINE = ReplicatedReplacingMergeTree('/clickhouse/{cluster}/tables/{database}/{table}', '{replica}', version) PRIMARY KEY (`type`, `key`) ORDER BY (`type`, `key`); +`type` String COMMENT 'Asset category (e.g. host, mac, network_device)', +`key` String COMMENT 'Unique asset key within its type (e.g. IP address, MAC address)', +`ifid` UInt8 COMMENT 'ntopng interface on which this asset was observed', +`ip` String DEFAULT '' COMMENT 'IP address of the asset (empty if not applicable)', +`mac` String COMMENT 'MAC address of the asset', +`vlan` UInt16 DEFAULT 0 COMMENT 'VLAN on which the asset was observed (0 if untagged)', +`network` UInt16 DEFAULT 0 COMMENT 'ntopng local-network ID the asset belongs to', +`name` String DEFAULT '' COMMENT 'Resolved hostname or user-defined name', +`device_type` UInt16 DEFAULT 0 COMMENT 'Device category/type (maps to ntopng DeviceType enum)', +`manufacturer` String DEFAULT '' COMMENT 'Hardware manufacturer derived from MAC OUI lookup', +`first_seen` DateTime COMMENT 'Timestamp when this asset was first observed by ntopng', +`last_seen` DateTime COMMENT 'Timestamp of the most recent observation of this asset', +`gateway_mac` String DEFAULT '' COMMENT 'MAC address of the gateway used to reach this asset', +`json_info` String DEFAULT '' -- A json containing all other info + COMMENT 'Additional asset metadata as a JSON blob (OS info, open ports, etc.)', +`version` UInt64 -- Used to not have duplicates + COMMENT 'Monotonically increasing version counter used by ReplacingMergeTree for deduplication', +`os_type` String DEFAULT '' COMMENT 'Operating system type detected for this asset', +`model` String DEFAULT '' COMMENT 'Hardware model string for this asset' +) ENGINE = ReplicatedReplacingMergeTree('/clickhouse/{cluster}/tables/{database}/{table}', '{replica}', version) PRIMARY KEY (`type`, `key`) ORDER BY (`type`, `key`) +COMMENT 'Network asset inventory: one row per discovered or imported asset (host, MAC address, network device). Uses ReplicatedReplacingMergeTree on version so that re-discovered assets update existing rows rather than creating duplicates. json_info holds additional metadata as a JSON blob.'; @ ALTER TABLE assets ADD COLUMN IF NOT EXISTS `os_type` String; @ @@ -1047,8 +1081,8 @@ SELECT FROM `flows` AS f LEFT JOIN `mitre_table_info` AS mitre ON (mitre.ENTITY_ID = 4 AND f.STATUS = mitre.ALERT_ID) -WHERE f.STATUS != 0 - AND f.IS_ALERT_DELETED != 1; +WHERE f.STATUS != 0 + AND f.IS_ALERT_DELETED != 1; @ @@ -1079,24 +1113,25 @@ SELECT 10 entity_id, interface_id, alert_id, alert_status, require_attention, ts @ CREATE TABLE IF NOT EXISTS `hourly_asn` ON CLUSTER '$CLUSTER' ( -`ID` UInt64, -`NTOPNG_INSTANCE_NAME` String, -`INTERFACE_ID` UInt16, -`IP_PROTOCOL_VERSION` UInt8, -`FIRST_SEEN` DateTime, -`LAST_SEEN` DateTime, -`SRC2DST_BYTES` UInt64, -`DST2SRC_BYTES` UInt64, -`TOTAL_BYTES` UInt64, -`SRC2DST_PACKETS` UInt32, -`DST2SRC_PACKETS` UInt32, -`SRC_ASN` UInt32, -`DST_ASN` UInt32, -`SRC_PEER_ASN` UInt32, -`DST_PEER_ASN` UInt32, -`PROBE_IP` UInt32, /* EXPORTER_IPV4_ADDRESS */ -`INPUT_SNMP` UInt32, -`OUTPUT_SNMP` UInt32 -) ENGINE = ReplicatedMergeTree('/clickhouse/{cluster}/tables/{database}/{table}', '{replica}') PARTITION BY toYYYYMMDD(FIRST_SEEN) ORDER BY (FIRST_SEEN, SRC_ASN, DST_ASN); +`ID` UInt64 COMMENT 'Unique row identifier', +`NTOPNG_INSTANCE_NAME` String COMMENT 'Name of the ntopng instance that generated this record', +`INTERFACE_ID` UInt16 COMMENT 'ntopng interface identifier on which this traffic was observed', +`IP_PROTOCOL_VERSION` UInt8 COMMENT 'IP version: 4 for IPv4, 6 for IPv6', +`FIRST_SEEN` DateTime COMMENT 'Start of the one-hour aggregation window', +`LAST_SEEN` DateTime COMMENT 'End of the one-hour aggregation window', +`SRC2DST_BYTES` UInt64 COMMENT 'Bytes from source ASN to destination ASN in this hour', +`DST2SRC_BYTES` UInt64 COMMENT 'Bytes from destination ASN to source ASN in this hour', +`TOTAL_BYTES` UInt64 COMMENT 'Total bytes between the two ASNs in this hour', +`SRC2DST_PACKETS` UInt32 COMMENT 'Packets from source ASN to destination ASN in this hour', +`DST2SRC_PACKETS` UInt32 COMMENT 'Packets from destination ASN to source ASN in this hour', +`SRC_ASN` UInt32 COMMENT 'Autonomous System Number of the source', +`DST_ASN` UInt32 COMMENT 'Autonomous System Number of the destination', +`SRC_PEER_ASN` UInt32 COMMENT 'BGP peer ASN upstream of the source (EXPORTER_IPV4_ADDRESS)', +`DST_PEER_ASN` UInt32 COMMENT 'BGP peer ASN upstream of the destination', +`PROBE_IP` UInt32 /* EXPORTER_IPV4_ADDRESS */ COMMENT 'IPv4 address of the NetFlow/IPFIX exporter; same as EXPORTER_IPV4_ADDRESS', +`INPUT_SNMP` UInt32 COMMENT 'SNMP input interface index from NetFlow/IPFIX', +`OUTPUT_SNMP` UInt32 COMMENT 'SNMP output interface index from NetFlow/IPFIX' +) ENGINE = ReplicatedMergeTree('/clickhouse/{cluster}/tables/{database}/{table}', '{replica}') PARTITION BY toYYYYMMDD(FIRST_SEEN) ORDER BY (FIRST_SEEN, SRC_ASN, DST_ASN) +COMMENT 'Hourly aggregated traffic statistics per source/destination ASN pair. Used for autonomous-system level traffic analysis and BGP peer analytics. Partitioned by day on FIRST_SEEN.'; @ ALTER TABLE `hourly_asn` ON CLUSTER '$CLUSTER' ADD COLUMN IF NOT EXISTS TOTAL_BYTES UInt64;