mirror of
https://github.com/rcourtman/Pulse.git
synced 2026-08-25 17:02:10 +00:00
Fix PBS backup attribution for VMIDs shared across clusters
v6.1.0's identity rework (eab73d245) made the VMID-only fallback refuse
to fire whenever a typed VMID exists on more than one PVE location.
Root-namespace snapshots with no matching comment then score zero for
every guest, so on setups with two clusters and overlapping VMIDs most
guests showed no backup at all - while PVE itself listed the backups
fine, because monitoring discards pbs-type storage contents entirely
whenever a direct PBS connection is configured.
Attribution is now evidence-driven instead of dropped:
Storage backup polling keeps a per-connection record of every snapshot
its own pbs-type storage listed (type, VMID, backup time) even though
the raw entries stay out of the PVE backup list. Which cluster listed a
snapshot is deterministic attribution, and it survives fully mirrored
clusters that share one datastore and token. The evidence is
monitoring-internal, cleared on instance retirement or when the storage
poll stops seeing pbs content, and never serialized into state payloads
or snapshots.
Guest backup-time sync and the recovery-point mapper additionally learn
each PBS submission source's cluster (owner token, datastore, PBS
instance - strongest first, scoped to the PBS instance) from the poll's
attributable snapshots, then resolve collision VMIDs whose snapshots
carry no evidence of their own. A source seen from several clusters is
not a discriminator, an unfamiliar component stops resolution rather
than deferring to weaker ones, and a snapshot decisively attributed to
another cluster is kept away from this one. Unattributable snapshots
still drop rather than guess.
Backup-age alert attribution no longer suffix-matches the subject ref's
connection label against guest locations. The label there is a PVE or
PBS instance name, not a PBS namespace, and loose matching could
cross-attribute clusters sharing a VMID; it now requires exact
normalized equality.
Reported in #1639 (two PVE clusters with PBS 4.0/4.1, VM 173 shown 974
days overdue despite valid verified backups).
Fixes #1639
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
This commit is contained in:
courtmanr@gmail.com
parent
ac43506e6e
commit
84dba861b5
19 changed files with 990 additions and 19 deletions
|
|
@ -410,6 +410,11 @@ update, profile rollout, command reachability, or fleet-control authority.
|
|||
24. `internal/config/host_continuity.go` shared with `monitoring`: the durable host identity, report-order watermark, and removal tombstone journal is jointly owned by agent lifecycle admission and monitoring report continuity.
|
||||
25. `internal/kubernetesagent/agent.go` shared with `monitoring`: the Kubernetes native agent runtime is both a monitoring inventory source and an agent lifecycle Pulse control-plane transport client.
|
||||
26. `internal/models/models.go` shared with `monitoring`: removed host-agent identity aliases and tombstone state are both agent lifecycle authority and monitoring runtime report state.
|
||||
The PVE-side PBS guest-backup confirmation evidence the same file carries
|
||||
is monitoring-owned attribution state for Proxmox backup freshness: it is
|
||||
internal-only, never serialized into state payloads or snapshots, and does
|
||||
not describe agent enrollment, report admission, tombstones,
|
||||
re-enrollment, or command authority.
|
||||
27. `internal/monitoring/monitor.go` shared with `monitoring`: monitor construction owns both monitoring runtime initialization and fail-closed agent lifecycle journal hydration before report admission.
|
||||
28. `internal/monitoring/monitor_agents.go` shared with `monitoring`: server-side Unified Agent report, removal, token binding, tombstone expiry, and re-enrollment semantics are jointly owned by agent lifecycle authority and monitoring ingest.
|
||||
29. `pkg/agents/host/report.go` shared with `monitoring`: the Unified Agent host report is both an agent lifecycle authored-state contract and a monitoring ingest contract for host maintenance posture.
|
||||
|
|
|
|||
|
|
@ -32,6 +32,13 @@ state must be explicitly true. Exit code 137 alone is only SIGKILL evidence;
|
|||
explicit false and unavailable/legacy OOM state both fail closed without an OOM
|
||||
alert. Recovery clears an existing OOM alert when the authoritative predicate
|
||||
is no longer true.
|
||||
Backup-age alert attribution of an unlinked recovery rollup treats the
|
||||
subject ref's namespace field as a connection label, not a PBS namespace: it
|
||||
may match a candidate guest's instance or node only by exact normalized
|
||||
equality. Suffix matching is reserved for real PBS namespaces inside the
|
||||
shared identity helpers; applying it to a PBS connection name
|
||||
cross-attributes clusters that share a VMID, so an unlinked backup whose
|
||||
label identifies no guest exactly stays on its generic rollup key.
|
||||
Availability incident and alert identity belongs to the source-owned
|
||||
`network-endpoint` check. Correlation may project probe evidence onto a matched
|
||||
machine, but it must not copy the check incident onto that machine or create a
|
||||
|
|
|
|||
|
|
@ -82,6 +82,22 @@ full backup point batch per poll cycle. Complete authoritative enumerations
|
|||
coalesce only within the same provider, ID-prefix, and instance scope; distinct
|
||||
source scopes and non-reconciling event batches remain FIFO so bounding memory
|
||||
does not discard independent recovery facts.
|
||||
PBS snapshot-to-guest attribution for VMIDs that exist on more than one PVE
|
||||
location is evidence-driven, never guessed. When the direct PBS connection is
|
||||
authoritative and pbs-type storage contents are dropped from the PVE backup
|
||||
list, storage backup polling must still harvest each listed snapshot as a
|
||||
per-connection guest confirmation (type, VMID, backup time): which cluster
|
||||
listed a snapshot through its own storage is the only deterministic
|
||||
attribution for collision VMIDs with no namespace or comment evidence. Guest
|
||||
backup-time sync consumes those confirmations first, then a
|
||||
submission-source mapping (owner token, datastore, PBS instance — scoped to
|
||||
the PBS instance, strongest first) learned from the same poll's attributable
|
||||
snapshots; a source component that was never positively attributed stops
|
||||
resolution rather than deferring to weaker components. Snapshots that remain
|
||||
unattributable stay dropped for colliding guests, and the confirmation
|
||||
evidence is monitoring-internal state: cleared when a PVE instance is
|
||||
retired or its storage poll returns no pbs-type content, and never
|
||||
serialized into state payloads or snapshots.
|
||||
Removed host-agent reconnect blocks are identity-scoped: matching may use the
|
||||
canonical host ID or token-qualified machine/hostname continuity, but must never
|
||||
block a distinct live host by hostname alone.
|
||||
|
|
|
|||
|
|
@ -5436,6 +5436,7 @@
|
|||
"internal/dockeragent/swarm_coverage_test.go",
|
||||
"internal/models/deepcopy_test.go",
|
||||
"internal/models/host_transient_expiry_test.go",
|
||||
"internal/models/issue1639_pbs_collision_test.go",
|
||||
"internal/models/metrics_types_test.go",
|
||||
"internal/models/state_host_test.go",
|
||||
"internal/monitoring/issue1595_collection_trust_test.go",
|
||||
|
|
@ -5572,6 +5573,7 @@
|
|||
"test_prefixes": [],
|
||||
"exact_files": [
|
||||
"internal/monitoring/monitor_alert_intent_test.go",
|
||||
"internal/monitoring/monitor_backups_dir_storage_test.go",
|
||||
"internal/monitoring/monitor_full_coverage_test.go",
|
||||
"internal/monitoring/pbs_protection_observation_test.go",
|
||||
"internal/monitoring/recovery_ingest_test.go",
|
||||
|
|
|
|||
|
|
@ -49,6 +49,15 @@ archive, backup task, recovery-point, restore, PBS, routing, or credential key.
|
|||
Search includes both presentation and native diagnostics, duplicate display
|
||||
values are allowed, and clearing an override deterministically restores the
|
||||
current native node name.
|
||||
Recovery-point mapping of PBS snapshots whose VMID exists on more than one
|
||||
PVE connection may link a snapshot only on positive evidence: namespace
|
||||
placement, guest-name label, or the submission-source mapping (owner token,
|
||||
datastore, PBS instance — strongest first, scoped to the PBS instance)
|
||||
learned from the same batch's attributable snapshots. A source component
|
||||
never positively attributed stops resolution instead of deferring to weaker
|
||||
components, and a snapshot whose source decisively belongs to a connection
|
||||
with no matching guest stays unlinked; it must never be linked to another
|
||||
cluster's guest with the same VMID.
|
||||
|
||||
|
||||
Physical-disk to node association is evidence-directional. A disk reported by a
|
||||
|
|
|
|||
|
|
@ -1645,9 +1645,12 @@ func TestCheckBackupsHandlesPbsOnlyGuests(t *testing.T) {
|
|||
|
||||
func TestCheckBackupsDisambiguatesWithNamespace(t *testing.T) {
|
||||
// Test that when multiple guests have the same VMID from different instances,
|
||||
// the namespace is used to match the backup to the correct guest.
|
||||
// This addresses issue #1095 where users have multiple PVE instances with
|
||||
// overlapping VMIDs and separate PBS instances backing them up.
|
||||
// the subject ref's connection label is used to match the backup to the
|
||||
// correct guest. This addresses issue #1095 where users have multiple PVE
|
||||
// instances with overlapping VMIDs and separate PBS instances backing
|
||||
// them up. The label must match a guest's instance or node exactly:
|
||||
// suffix matching let a PBS instance name cross-attribute clusters that
|
||||
// share a VMID (#1639), so "nat" no longer matches "pve-nat".
|
||||
m := newTestManager(t)
|
||||
m.ClearActiveAlerts()
|
||||
|
||||
|
|
@ -1690,13 +1693,14 @@ func TestCheckBackupsDisambiguatesWithNamespace(t *testing.T) {
|
|||
},
|
||||
}
|
||||
|
||||
// PBS backup with namespace "nat" should match the "pve-nat" instance
|
||||
// A subject ref labeled with the exact connection name matches the
|
||||
// "pve-nat" instance.
|
||||
rollups := []recovery.ProtectionRollup{
|
||||
{
|
||||
RollupID: "ext:pbs-nat-100",
|
||||
SubjectRef: &recovery.ExternalRef{
|
||||
Type: "proxmox-vm",
|
||||
Namespace: "nat", // namespace "nat" should match the pve-nat location
|
||||
Namespace: "pve-nat", // exact connection label for the pve-nat location
|
||||
Name: "100",
|
||||
ID: "100",
|
||||
},
|
||||
|
|
@ -1976,6 +1980,86 @@ func TestCheckBackupsVMIDCollisionNoNamespace(t *testing.T) {
|
|||
}
|
||||
}
|
||||
|
||||
// TestCheckBackupsIssue1639PBSInstanceLabelDoesNotCrossAttribute verifies
|
||||
// that a PBS connection name that merely suffix-matches one guest's
|
||||
// instance name does not attribute the backup to that guest. The subject
|
||||
// ref of an unlinked PBS rollup carries the PBS instance name in Namespace,
|
||||
// and loose matching let it cross-attribute clusters sharing a VMID.
|
||||
func TestCheckBackupsIssue1639PBSInstanceLabelDoesNotCrossAttribute(t *testing.T) {
|
||||
m := newTestManager(t)
|
||||
m.ClearActiveAlerts()
|
||||
|
||||
m.mu.Lock()
|
||||
m.config.Enabled = true
|
||||
m.config.BackupDefaults = BackupAlertConfig{
|
||||
Enabled: true,
|
||||
WarningDays: 3,
|
||||
CriticalDays: 5,
|
||||
}
|
||||
m.mu.Unlock()
|
||||
|
||||
now := time.Now()
|
||||
|
||||
guestsByKey := map[string]GuestLookup{
|
||||
"pve-nat-node2-100": {
|
||||
ResourceID: "qemu/100",
|
||||
Name: "webserver-nat",
|
||||
Instance: "pve-nat",
|
||||
Node: "node2",
|
||||
Type: "qemu",
|
||||
VMID: 100,
|
||||
},
|
||||
"pve-main-node1-100": {
|
||||
ResourceID: "qemu/100",
|
||||
Name: "webserver-main",
|
||||
Instance: "pve-main",
|
||||
Node: "node1",
|
||||
Type: "qemu",
|
||||
VMID: 100,
|
||||
},
|
||||
}
|
||||
guestsByVMID := map[string][]GuestLookup{
|
||||
"100": {
|
||||
guestsByKey["pve-nat-node2-100"],
|
||||
guestsByKey["pve-main-node1-100"],
|
||||
},
|
||||
}
|
||||
|
||||
// "nat" is the PBS connection name, not a namespace. It suffix-matches
|
||||
// "pve-nat" but identifies neither guest; the alert must stay on the
|
||||
// generic rollup key instead of guessing.
|
||||
rollups := []recovery.ProtectionRollup{
|
||||
{
|
||||
RollupID: "ext:pbs-nat-100",
|
||||
SubjectRef: &recovery.ExternalRef{
|
||||
Type: "proxmox-vm",
|
||||
Namespace: "nat",
|
||||
Name: "100",
|
||||
ID: "100",
|
||||
},
|
||||
LastSuccessAt: ptrTime(now.Add(-6 * 24 * time.Hour)),
|
||||
LastOutcome: recovery.OutcomeSuccess,
|
||||
Providers: []recovery.Provider{recovery.ProviderProxmoxPBS},
|
||||
},
|
||||
}
|
||||
|
||||
m.CheckBackups(rollups, guestsByKey, guestsByVMID)
|
||||
|
||||
m.mu.RLock()
|
||||
defer m.mu.RUnlock()
|
||||
|
||||
if _, exists := testLookupActiveAlert(t, m, "backup-age-pve-nat-node2-100"); exists {
|
||||
t.Fatal("PBS connection label cross-attributed the backup to pve-nat")
|
||||
}
|
||||
var keys []string
|
||||
for storageKey, active := range m.activeAlerts {
|
||||
keys = append(keys, effectiveAlertID(active, storageKey))
|
||||
}
|
||||
if len(keys) != 1 || !strings.Contains(keys[0], "ext-pbs-nat-100") {
|
||||
t.Fatalf("expected a single generic rollup-keyed alert, found keys: %v", keys)
|
||||
}
|
||||
}
|
||||
|
||||
func TestCheckBackupsTypeMismatchDoesNotAttributeToWrongGuest(t *testing.T) {
|
||||
m := newTestManager(t)
|
||||
m.ClearActiveAlerts()
|
||||
|
|
|
|||
|
|
@ -686,17 +686,23 @@ func (m *Manager) CheckBackupsWithInventory(
|
|||
if len(guests) == 1 {
|
||||
info = guests[0]
|
||||
} else if len(guests) > 1 && strings.TrimSpace(ref.Namespace) != "" {
|
||||
// ref.Namespace carries a connection label here (the
|
||||
// PVE or PBS instance name from the mapper), not a
|
||||
// PBS namespace, so it must match a guest's location
|
||||
// exactly. Suffix matching a PBS instance name
|
||||
// against node names cross-attributes clusters that
|
||||
// share a VMID (#1639).
|
||||
bestScore := 0
|
||||
matchedMultiple := false
|
||||
for _, g := range guests {
|
||||
score := proxmoxidentity.BackupGuestMatchScore(
|
||||
ref.Namespace,
|
||||
ref.Name,
|
||||
vmidStr,
|
||||
g.Name,
|
||||
g.Instance,
|
||||
g.Node,
|
||||
)
|
||||
score := 0
|
||||
if proxmoxidentity.LocationLabelsEqual(ref.Namespace, g.Instance) ||
|
||||
proxmoxidentity.LocationLabelsEqual(ref.Namespace, g.Node) {
|
||||
score += 10
|
||||
}
|
||||
if proxmoxidentity.BackupCommentMatchesGuestName(ref.Name, vmidStr, g.Name) {
|
||||
score += 5
|
||||
}
|
||||
if score > bestScore {
|
||||
bestScore = score
|
||||
info = g
|
||||
|
|
|
|||
238
internal/models/issue1639_pbs_collision_test.go
Normal file
238
internal/models/issue1639_pbs_collision_test.go
Normal file
|
|
@ -0,0 +1,238 @@
|
|||
package models
|
||||
|
||||
import (
|
||||
"testing"
|
||||
"time"
|
||||
)
|
||||
|
||||
// Issue #1639: two PVE clusters with the same VMID, root-namespace PBS
|
||||
// snapshots, and no comment left both guests with zero LastBackup because
|
||||
// the ambiguity guard disabled the VMID-only fallback and the scorer never
|
||||
// consulted the snapshots' provenance (owner token, datastore, PBS
|
||||
// instance) or the PVE-side storage listings.
|
||||
|
||||
// TestIssue1639PBSCollisionResolvedBySubmissionSource verifies that when
|
||||
// each cluster's submission source is learnable from its other,
|
||||
// non-colliding guests, root-namespace comment-less snapshots for a
|
||||
// colliding VMID are attributed to the correct cluster.
|
||||
func TestIssue1639PBSCollisionResolvedBySubmissionSource(t *testing.T) {
|
||||
state := NewState()
|
||||
|
||||
now := time.Now()
|
||||
backupTimeA := now.Add(-2 * time.Hour)
|
||||
backupTimeB := now.Add(-3 * time.Hour)
|
||||
uniqueTimeA := now.Add(-26 * time.Hour)
|
||||
uniqueTimeB := now.Add(-27 * time.Hour)
|
||||
|
||||
state.UpdateVMs([]VM{
|
||||
// VMID 173 exists on both clusters — the reported collision.
|
||||
{VMID: 173, Name: "web-a", Instance: "cluster-a", Node: "pve-a1"},
|
||||
{VMID: 173, Name: "web-b", Instance: "cluster-b", Node: "pve-b1"},
|
||||
// Each cluster also has a unique VMID whose snapshots teach the
|
||||
// learner that cluster's owner token and datastore.
|
||||
{VMID: 100, Name: "db-a", Instance: "cluster-a", Node: "pve-a1"},
|
||||
{VMID: 150, Name: "db-b", Instance: "cluster-b", Node: "pve-b1"},
|
||||
})
|
||||
|
||||
state.mu.Lock()
|
||||
state.PBSBackups = []PBSBackup{
|
||||
// Teaching snapshots: unique VMIDs, root namespace, no comment.
|
||||
{ID: "a-100", VMID: "100", BackupType: "vm", BackupTime: uniqueTimeA,
|
||||
Instance: "pbs-main", Datastore: "store-a", Owner: "cluster-a@pbs!token"},
|
||||
{ID: "b-150", VMID: "150", BackupType: "vm", BackupTime: uniqueTimeB,
|
||||
Instance: "pbs-main", Datastore: "store-b", Owner: "cluster-b@pbs!token"},
|
||||
// Colliding VMID with no namespace/comment evidence, distinct
|
||||
// owner and datastore per cluster.
|
||||
{ID: "a-173", VMID: "173", BackupType: "vm", BackupTime: backupTimeA,
|
||||
Instance: "pbs-main", Datastore: "store-a", Owner: "cluster-a@pbs!token"},
|
||||
{ID: "b-173", VMID: "173", BackupType: "vm", BackupTime: backupTimeB,
|
||||
Instance: "pbs-main", Datastore: "store-b", Owner: "cluster-b@pbs!token"},
|
||||
}
|
||||
state.mu.Unlock()
|
||||
|
||||
state.SyncGuestBackupTimes()
|
||||
snapshot := state.GetSnapshot()
|
||||
|
||||
got := map[string]time.Time{}
|
||||
for _, vm := range snapshot.VMs {
|
||||
if vm.VMID == 173 {
|
||||
got[vm.Instance] = vm.LastBackup
|
||||
}
|
||||
}
|
||||
if !got["cluster-a"].Equal(backupTimeA) {
|
||||
t.Errorf("cluster-a VM 173 LastBackup = %v, want its own snapshot %v", got["cluster-a"], backupTimeA)
|
||||
}
|
||||
if !got["cluster-b"].Equal(backupTimeB) {
|
||||
t.Errorf("cluster-b VM 173 LastBackup = %v, want its own snapshot %v", got["cluster-b"], backupTimeB)
|
||||
}
|
||||
}
|
||||
|
||||
// TestIssue1639PBSCollisionResolvedByDistinctPBSInstance verifies the
|
||||
// weakest source component: each cluster pushes to its own PBS instance,
|
||||
// same datastore name, no owner reported.
|
||||
func TestIssue1639PBSCollisionResolvedByDistinctPBSInstance(t *testing.T) {
|
||||
state := NewState()
|
||||
|
||||
now := time.Now()
|
||||
backupTimeA := now.Add(-2 * time.Hour)
|
||||
backupTimeB := now.Add(-3 * time.Hour)
|
||||
|
||||
state.UpdateVMs([]VM{
|
||||
{VMID: 173, Name: "web-a", Instance: "cluster-a", Node: "pve-a1"},
|
||||
{VMID: 173, Name: "web-b", Instance: "cluster-b", Node: "pve-b1"},
|
||||
{VMID: 100, Name: "db-a", Instance: "cluster-a", Node: "pve-a1"},
|
||||
{VMID: 150, Name: "db-b", Instance: "cluster-b", Node: "pve-b1"},
|
||||
})
|
||||
|
||||
state.mu.Lock()
|
||||
state.PBSBackups = []PBSBackup{
|
||||
{ID: "a-100", VMID: "100", BackupType: "vm", BackupTime: now.Add(-26 * time.Hour), Instance: "pbs-one", Datastore: "backups"},
|
||||
{ID: "b-150", VMID: "150", BackupType: "vm", BackupTime: now.Add(-27 * time.Hour), Instance: "pbs-two", Datastore: "backups"},
|
||||
{ID: "a-173", VMID: "173", BackupType: "vm", BackupTime: backupTimeA, Instance: "pbs-one", Datastore: "backups"},
|
||||
{ID: "b-173", VMID: "173", BackupType: "vm", BackupTime: backupTimeB, Instance: "pbs-two", Datastore: "backups"},
|
||||
}
|
||||
state.mu.Unlock()
|
||||
|
||||
state.SyncGuestBackupTimes()
|
||||
snapshot := state.GetSnapshot()
|
||||
|
||||
for _, vm := range snapshot.VMs {
|
||||
switch {
|
||||
case vm.VMID == 173 && vm.Instance == "cluster-a":
|
||||
if !vm.LastBackup.Equal(backupTimeA) {
|
||||
t.Errorf("cluster-a VM 173 LastBackup = %v, want %v", vm.LastBackup, backupTimeA)
|
||||
}
|
||||
case vm.VMID == 173 && vm.Instance == "cluster-b":
|
||||
if !vm.LastBackup.Equal(backupTimeB) {
|
||||
t.Errorf("cluster-b VM 173 LastBackup = %v, want %v", vm.LastBackup, backupTimeB)
|
||||
}
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
// TestIssue1639PBSCollisionSharedSourceStaysDropped verifies the guard is
|
||||
// not weakened: when both clusters push through the same owner, datastore,
|
||||
// and PBS instance and no PVE-side evidence exists, the snapshots stay
|
||||
// unattributed rather than being guessed onto a cluster.
|
||||
func TestIssue1639PBSCollisionSharedSourceStaysDropped(t *testing.T) {
|
||||
state := NewState()
|
||||
|
||||
now := time.Now()
|
||||
|
||||
state.UpdateVMs([]VM{
|
||||
{VMID: 173, Name: "web-a", Instance: "cluster-a", Node: "pve-a1"},
|
||||
{VMID: 173, Name: "web-b", Instance: "cluster-b", Node: "pve-b1"},
|
||||
{VMID: 100, Name: "db-a", Instance: "cluster-a", Node: "pve-a1"},
|
||||
{VMID: 150, Name: "db-b", Instance: "cluster-b", Node: "pve-b1"},
|
||||
})
|
||||
|
||||
state.mu.Lock()
|
||||
state.PBSBackups = []PBSBackup{
|
||||
// Both clusters share one submission source, so nothing about the
|
||||
// colliding snapshots discriminates between them.
|
||||
{ID: "a-100", VMID: "100", BackupType: "vm", BackupTime: now.Add(-26 * time.Hour), Instance: "pbs-main", Datastore: "backups", Owner: "shared@pbs!token"},
|
||||
{ID: "b-150", VMID: "150", BackupType: "vm", BackupTime: now.Add(-27 * time.Hour), Instance: "pbs-main", Datastore: "backups", Owner: "shared@pbs!token"},
|
||||
{ID: "x-173", VMID: "173", BackupType: "vm", BackupTime: now.Add(-2 * time.Hour), Instance: "pbs-main", Datastore: "backups", Owner: "shared@pbs!token"},
|
||||
}
|
||||
state.mu.Unlock()
|
||||
|
||||
state.SyncGuestBackupTimes()
|
||||
snapshot := state.GetSnapshot()
|
||||
|
||||
for _, vm := range snapshot.VMs {
|
||||
if vm.VMID == 173 && !vm.LastBackup.IsZero() {
|
||||
t.Errorf("VM 173 on %s should stay unattributed with a shared submission source, got %v", vm.Instance, vm.LastBackup)
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
// TestIssue1639PBSCollisionResolvedByPVEStorageConfirmation verifies the
|
||||
// deterministic path: even with identical submission sources (fully
|
||||
// mirrored clusters pushing to one datastore with one token), each PVE
|
||||
// connection's own pbs-type storage listing attributes its snapshots.
|
||||
func TestIssue1639PBSCollisionResolvedByPVEStorageConfirmation(t *testing.T) {
|
||||
state := NewState()
|
||||
|
||||
now := time.Now()
|
||||
backupTimeA := now.Add(-2 * time.Hour).Truncate(time.Second)
|
||||
backupTimeB := now.Add(-3 * time.Hour).Truncate(time.Second)
|
||||
|
||||
state.UpdateVMs([]VM{
|
||||
{VMID: 173, Name: "web", Instance: "cluster-a", Node: "pve-a1"},
|
||||
{VMID: 173, Name: "web", Instance: "cluster-b", Node: "pve-b1"},
|
||||
})
|
||||
|
||||
state.mu.Lock()
|
||||
state.PBSBackups = []PBSBackup{
|
||||
{ID: "a-173", VMID: "173", BackupType: "vm", BackupTime: backupTimeA, Instance: "pbs-main", Datastore: "backups", Owner: "shared@pbs!token"},
|
||||
{ID: "b-173", VMID: "173", BackupType: "vm", BackupTime: backupTimeB, Instance: "pbs-main", Datastore: "backups", Owner: "shared@pbs!token"},
|
||||
}
|
||||
state.mu.Unlock()
|
||||
|
||||
state.UpdatePBSGuestConfirmationsForInstance("cluster-a", []PBSGuestConfirmation{
|
||||
{BackupType: "vm", VMID: 173, Time: backupTimeA.Unix()},
|
||||
})
|
||||
state.UpdatePBSGuestConfirmationsForInstance("cluster-b", []PBSGuestConfirmation{
|
||||
{BackupType: "vm", VMID: 173, Time: backupTimeB.Unix()},
|
||||
})
|
||||
|
||||
state.SyncGuestBackupTimes()
|
||||
snapshot := state.GetSnapshot()
|
||||
|
||||
for _, vm := range snapshot.VMs {
|
||||
switch vm.Instance {
|
||||
case "cluster-a":
|
||||
if !vm.LastBackup.Equal(backupTimeA) {
|
||||
t.Errorf("cluster-a VM 173 LastBackup = %v, want confirmed snapshot %v", vm.LastBackup, backupTimeA)
|
||||
}
|
||||
case "cluster-b":
|
||||
if !vm.LastBackup.Equal(backupTimeB) {
|
||||
t.Errorf("cluster-b VM 173 LastBackup = %v, want confirmed snapshot %v", vm.LastBackup, backupTimeB)
|
||||
}
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
// TestIssue1639PBSCollisionForeignSourceRejected verifies negative
|
||||
// attribution: a snapshot whose source decisively belongs to another
|
||||
// cluster must not reach this cluster's guest even though the guest has no
|
||||
// snapshot of its own.
|
||||
func TestIssue1639PBSCollisionForeignSourceRejected(t *testing.T) {
|
||||
state := NewState()
|
||||
|
||||
now := time.Now()
|
||||
|
||||
state.UpdateVMs([]VM{
|
||||
{VMID: 173, Name: "web-a", Instance: "cluster-a", Node: "pve-a1"},
|
||||
{VMID: 173, Name: "web-b", Instance: "cluster-b", Node: "pve-b1"},
|
||||
{VMID: 100, Name: "db-a", Instance: "cluster-a", Node: "pve-a1"},
|
||||
})
|
||||
|
||||
state.mu.Lock()
|
||||
state.PBSBackups = []PBSBackup{
|
||||
// Only cluster-a's source is learnable, and the only 173 snapshot
|
||||
// belongs to it. cluster-b must stay at zero.
|
||||
{ID: "a-100", VMID: "100", BackupType: "vm", BackupTime: now.Add(-26 * time.Hour), Instance: "pbs-main", Datastore: "store-a", Owner: "cluster-a@pbs!token"},
|
||||
{ID: "a-173", VMID: "173", BackupType: "vm", BackupTime: now.Add(-2 * time.Hour), Instance: "pbs-main", Datastore: "store-a", Owner: "cluster-a@pbs!token"},
|
||||
}
|
||||
state.mu.Unlock()
|
||||
|
||||
state.SyncGuestBackupTimes()
|
||||
snapshot := state.GetSnapshot()
|
||||
|
||||
for _, vm := range snapshot.VMs {
|
||||
if vm.VMID != 173 {
|
||||
continue
|
||||
}
|
||||
switch vm.Instance {
|
||||
case "cluster-a":
|
||||
if vm.LastBackup.IsZero() {
|
||||
t.Error("cluster-a VM 173 should get its own snapshot")
|
||||
}
|
||||
case "cluster-b":
|
||||
if !vm.LastBackup.IsZero() {
|
||||
t.Errorf("cluster-b VM 173 must not inherit cluster-a's snapshot, got %v", vm.LastBackup)
|
||||
}
|
||||
}
|
||||
}
|
||||
}
|
||||
|
|
@ -209,3 +209,31 @@ func TestNativePoolHealthEvidenceNormalizesWithoutChangingIdentity(t *testing.T)
|
|||
t.Fatalf("normalized Ceph evidence = %+v", cluster)
|
||||
}
|
||||
}
|
||||
|
||||
// TestPBSGuestConfirmationEvidenceStaysOutOfSerializedState guards the
|
||||
// boundary the #1639 fix introduced: PVE-side PBS snapshot confirmations
|
||||
// are monitoring-internal attribution evidence on State. They must never
|
||||
// leak into serialized state or snapshots, where they would read as an
|
||||
// agent- or API-facing contract.
|
||||
func TestPBSGuestConfirmationEvidenceStaysOutOfSerializedState(t *testing.T) {
|
||||
state := NewState()
|
||||
state.UpdatePBSGuestConfirmationsForInstance("cluster-a", []PBSGuestConfirmation{
|
||||
{BackupType: "vm", VMID: 173, Time: time.Now().Unix()},
|
||||
})
|
||||
|
||||
payload, err := json.Marshal(state)
|
||||
if err != nil {
|
||||
t.Fatalf("marshal state: %v", err)
|
||||
}
|
||||
if strings.Contains(string(payload), "pbsGuestConfirmations") || strings.Contains(string(payload), "PBSGuestConfirmation") {
|
||||
t.Fatal("PBS guest confirmation evidence must not serialize with state")
|
||||
}
|
||||
|
||||
snapshotPayload, err := json.Marshal(state.GetSnapshot())
|
||||
if err != nil {
|
||||
t.Fatalf("marshal snapshot: %v", err)
|
||||
}
|
||||
if strings.Contains(string(snapshotPayload), "pbsGuestConfirmations") || strings.Contains(string(snapshotPayload), "PBSGuestConfirmation") {
|
||||
t.Fatal("PBS guest confirmation evidence must not appear in snapshots")
|
||||
}
|
||||
}
|
||||
|
|
|
|||
|
|
@ -48,6 +48,26 @@ type State struct {
|
|||
TemperatureMonitoringEnabled bool `json:"temperatureMonitoringEnabled"`
|
||||
PVETagColors map[string]string `json:"pveTagColors,omitempty"`
|
||||
PVETagStyles map[string]PVETagStyle `json:"pveTagStyles,omitempty"`
|
||||
|
||||
// pbsGuestConfirmations records, per PVE connection, the (type, vmid,
|
||||
// time) triples that connection's own pbs-type storage listings
|
||||
// reported. When direct PBS polling is authoritative the raw storage
|
||||
// contents are not kept as StorageBackups, but which cluster listed a
|
||||
// snapshot is the only deterministic way to attribute a PBS snapshot
|
||||
// whose VMID exists on several clusters (#1639). Internal evidence only:
|
||||
// never serialized and never part of snapshots.
|
||||
pbsGuestConfirmations map[string]map[PBSGuestConfirmation]struct{}
|
||||
}
|
||||
|
||||
// PBSGuestConfirmation identifies one PBS snapshot as seen through a PVE
|
||||
// connection's own pbs-type storage content listing. BackupType uses PBS
|
||||
// nomenclature ("vm" or "ct") and Time is the snapshot's backup time in
|
||||
// unix seconds, which matches PBSBackup.BackupTime exactly because both
|
||||
// derive from the snapshot identifier.
|
||||
type PBSGuestConfirmation struct {
|
||||
BackupType string
|
||||
VMID int
|
||||
Time int64
|
||||
}
|
||||
|
||||
// PVETagStyle is the frontend-facing Proxmox tag style for one PVE instance.
|
||||
|
|
@ -4107,6 +4127,36 @@ func backupKey(instance string, vmid int) string {
|
|||
return instance + "-" + strconv.Itoa(vmid)
|
||||
}
|
||||
|
||||
// UpdatePBSGuestConfirmationsForInstance replaces the PVE-side PBS snapshot
|
||||
// evidence for one connection. Pass an empty slice when the connection's
|
||||
// storage poll saw no pbs-type storage content so stale evidence cannot
|
||||
// outlive the snapshots it described.
|
||||
func (s *State) UpdatePBSGuestConfirmationsForInstance(instanceName string, confirmations []PBSGuestConfirmation) {
|
||||
s.mu.Lock()
|
||||
defer s.mu.Unlock()
|
||||
|
||||
if s.pbsGuestConfirmations == nil {
|
||||
s.pbsGuestConfirmations = make(map[string]map[PBSGuestConfirmation]struct{})
|
||||
}
|
||||
if len(confirmations) == 0 {
|
||||
delete(s.pbsGuestConfirmations, instanceName)
|
||||
return
|
||||
}
|
||||
set := make(map[PBSGuestConfirmation]struct{}, len(confirmations))
|
||||
for _, confirmation := range confirmations {
|
||||
confirmation.BackupType = strings.ToLower(strings.TrimSpace(confirmation.BackupType))
|
||||
if (confirmation.BackupType != "vm" && confirmation.BackupType != "ct") || confirmation.VMID <= 0 || confirmation.Time <= 0 {
|
||||
continue
|
||||
}
|
||||
set[confirmation] = struct{}{}
|
||||
}
|
||||
if len(set) == 0 {
|
||||
delete(s.pbsGuestConfirmations, instanceName)
|
||||
return
|
||||
}
|
||||
s.pbsGuestConfirmations[instanceName] = set
|
||||
}
|
||||
|
||||
// SyncGuestBackupTimes updates LastBackup on VMs and Containers from storage backups and PBS backups.
|
||||
// Call this after updating storage backups or PBS backups to ensure guest backup indicators are accurate.
|
||||
// Matching is done by instance+VMID to prevent cross-instance VMID collisions.
|
||||
|
|
@ -4155,7 +4205,13 @@ func (s *State) SyncGuestBackupTimes() {
|
|||
// Build a set of typed VMIDs that appear on more than one PVE location.
|
||||
// When a typed VMID is ambiguous, we must not fall back to VMID-only matching
|
||||
// because we can't tell which guest the backup belongs to.
|
||||
type pbsSubjectGuest struct {
|
||||
instance string
|
||||
node string
|
||||
name string
|
||||
}
|
||||
subjectLocations := make(map[pbsSubjectKey]map[string]struct{})
|
||||
subjectGuests := make(map[pbsSubjectKey][]pbsSubjectGuest)
|
||||
for i := range s.VMs {
|
||||
key := pbsSubjectKey{backupType: "vm", vmid: s.VMs[i].VMID}
|
||||
m, ok := subjectLocations[key]
|
||||
|
|
@ -4164,6 +4220,7 @@ func (s *State) SyncGuestBackupTimes() {
|
|||
subjectLocations[key] = m
|
||||
}
|
||||
m[backupKey(s.VMs[i].Instance, s.VMs[i].VMID)+"@"+s.VMs[i].Node] = struct{}{}
|
||||
subjectGuests[key] = append(subjectGuests[key], pbsSubjectGuest{instance: s.VMs[i].Instance, node: s.VMs[i].Node, name: s.VMs[i].Name})
|
||||
}
|
||||
for i := range s.Containers {
|
||||
key := pbsSubjectKey{backupType: "ct", vmid: s.Containers[i].VMID}
|
||||
|
|
@ -4173,6 +4230,7 @@ func (s *State) SyncGuestBackupTimes() {
|
|||
subjectLocations[key] = m
|
||||
}
|
||||
m[backupKey(s.Containers[i].Instance, s.Containers[i].VMID)+"@"+s.Containers[i].Node] = struct{}{}
|
||||
subjectGuests[key] = append(subjectGuests[key], pbsSubjectGuest{instance: s.Containers[i].Instance, node: s.Containers[i].Node, name: s.Containers[i].Name})
|
||||
}
|
||||
subjectIsAmbiguous := make(map[pbsSubjectKey]bool)
|
||||
for key, locations := range subjectLocations {
|
||||
|
|
@ -4181,6 +4239,57 @@ func (s *State) SyncGuestBackupTimes() {
|
|||
}
|
||||
}
|
||||
|
||||
// PVE-side confirmations deterministically attribute a PBS snapshot to
|
||||
// the connection whose own pbs-type storage listed it. Index them by
|
||||
// snapshot identity, keeping the set of confirming connections so a
|
||||
// snapshot listed by several connections (shouldn't happen, but trust
|
||||
// nothing) stays inconclusive.
|
||||
confirmingInstances := make(map[PBSGuestConfirmation]map[string]struct{})
|
||||
for instanceName, confirmations := range s.pbsGuestConfirmations {
|
||||
for confirmation := range confirmations {
|
||||
set, ok := confirmingInstances[confirmation]
|
||||
if !ok {
|
||||
set = make(map[string]struct{})
|
||||
confirmingInstances[confirmation] = set
|
||||
}
|
||||
set[instanceName] = struct{}{}
|
||||
}
|
||||
}
|
||||
|
||||
// Learn which PVE connection each PBS submission source (owner token,
|
||||
// datastore, PBS instance) belongs to from the backups that CAN be
|
||||
// attributed — unique typed VMIDs, or placement/name matches. The
|
||||
// learned mapping then resolves collision-VMID snapshots that carry no
|
||||
// namespace or comment evidence of their own (#1639).
|
||||
sourceLearner := proxmoxidentity.NewPBSSourceLearner()
|
||||
for key, backups := range pbsBackupsBySubject {
|
||||
guests := subjectGuests[key]
|
||||
if len(guests) == 0 {
|
||||
continue
|
||||
}
|
||||
for _, backup := range backups {
|
||||
attributed := ""
|
||||
if !subjectIsAmbiguous[key] {
|
||||
attributed = guests[0].instance
|
||||
} else {
|
||||
matched := make(map[string]struct{})
|
||||
for _, guest := range guests {
|
||||
if proxmoxidentity.BackupGuestMatchScore(backup.Namespace, backup.Comment, backup.VMID, guest.name, guest.instance, guest.node) > 0 {
|
||||
matched[guest.instance] = struct{}{}
|
||||
}
|
||||
}
|
||||
if len(matched) == 1 {
|
||||
for instance := range matched {
|
||||
attributed = instance
|
||||
}
|
||||
}
|
||||
}
|
||||
if attributed != "" {
|
||||
sourceLearner.Observe(backup.Instance, backup.Datastore, backup.Owner, attributed)
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
// findBestPBSBackup finds the best PBS backup for a given typed VMID and guest location.
|
||||
// Placement and guest-name matches are preferred over VMID-only fallback.
|
||||
// Returns zero time if no suitable backup found.
|
||||
|
|
@ -4203,8 +4312,26 @@ func (s *State) SyncGuestBackupTimes() {
|
|||
instance,
|
||||
node,
|
||||
)
|
||||
if score == 0 && !subjectIsAmbiguous[subjectKey] {
|
||||
score = 1
|
||||
if score == 0 {
|
||||
if !subjectIsAmbiguous[subjectKey] {
|
||||
score = 1
|
||||
} else if confirmers := confirmingInstances[PBSGuestConfirmation{BackupType: backupType, VMID: vmid, Time: backup.BackupTime.Unix()}]; len(confirmers) == 1 {
|
||||
// Exactly one connection's own PBS storage listed this
|
||||
// snapshot — it belongs to that cluster, full stop.
|
||||
if _, ours := confirmers[instance]; ours {
|
||||
score = 2
|
||||
}
|
||||
} else {
|
||||
// No usable storage-view evidence (unconfirmed, or a
|
||||
// shared datastore mount several clusters list); fall
|
||||
// back to the learned submission-source mapping. A
|
||||
// decisive attribution to another cluster keeps the
|
||||
// snapshot away from this guest; an inconclusive one
|
||||
// keeps the pre-#1639 drop.
|
||||
if attributed, decisive := sourceLearner.Resolve(backup.Instance, backup.Datastore, backup.Owner); decisive && attributed == instance {
|
||||
score = 1
|
||||
}
|
||||
}
|
||||
}
|
||||
if score <= 0 {
|
||||
continue
|
||||
|
|
|
|||
|
|
@ -1159,7 +1159,9 @@ func TestSyncGuestBackupTimesClearsStaleBackupWhenCurrentEvidenceDisappears(t *t
|
|||
|
||||
// TestSyncGuestBackupTimesVMIDCollisionNonMatchingNamespace verifies that when the same VMID
|
||||
// exists on multiple PVE instances and a PBS backup namespace matches neither, both guests
|
||||
// get zero LastBackup instead of a false positive.
|
||||
// get zero LastBackup instead of a false positive. The backup's submission
|
||||
// source (owner/datastore) carries no learned attribution and no PVE-side
|
||||
// storage listing confirms it, so the #1639 evidence paths must not fire.
|
||||
func TestSyncGuestBackupTimesVMIDCollisionNonMatchingNamespace(t *testing.T) {
|
||||
state := NewState()
|
||||
|
||||
|
|
@ -1182,6 +1184,8 @@ func TestSyncGuestBackupTimesVMIDCollisionNonMatchingNamespace(t *testing.T) {
|
|||
BackupType: "vm",
|
||||
BackupTime: backupTime,
|
||||
Instance: "pbs-main",
|
||||
Datastore: "backups",
|
||||
Owner: "unlearned@pbs!token",
|
||||
},
|
||||
}
|
||||
state.mu.Unlock()
|
||||
|
|
@ -1199,7 +1203,10 @@ func TestSyncGuestBackupTimesVMIDCollisionNonMatchingNamespace(t *testing.T) {
|
|||
|
||||
// TestSyncGuestBackupTimesVMIDCollisionEmptyNamespace verifies that when the same VMID
|
||||
// exists on multiple PVE instances and a PBS backup has no namespace, both guests
|
||||
// get zero LastBackup.
|
||||
// get zero LastBackup. This is the genuinely unattributable core of #1639:
|
||||
// with a single snapshot, no learnable submission source, and no PVE-side
|
||||
// confirmation, guessing either guest would be a false positive. The
|
||||
// resolvable variants live in issue1639_pbs_collision_test.go.
|
||||
func TestSyncGuestBackupTimesVMIDCollisionEmptyNamespace(t *testing.T) {
|
||||
state := NewState()
|
||||
|
||||
|
|
|
|||
|
|
@ -149,6 +149,7 @@ func (m *Monitor) backupInventoryScopeForAlerts() *alerts.BackupInventoryScope {
|
|||
func (m *Monitor) pollStorageBackupsWithNodes(ctx context.Context, instanceName string, client PVEClientInterface, nodes []proxmox.Node, nodeEffectiveStatus map[string]string) {
|
||||
|
||||
var allBackups []models.StorageBackup
|
||||
var pbsConfirmations []models.PBSGuestConfirmation
|
||||
hasPBSDirectConnection := m.config != nil && len(m.config.PBSInstances) > 0
|
||||
seenVolids := make(map[string]bool) // Track seen volume IDs to avoid duplicates
|
||||
hadSuccessfulNode := false // Track if at least one node responded successfully
|
||||
|
|
@ -306,6 +307,19 @@ func (m *Monitor) pollStorageBackupsWithNodes(ctx context.Context, instanceName
|
|||
// connection was also configured (#1592).
|
||||
isPBSStorage := storage.Type == "pbs"
|
||||
if isPBSStorage && hasPBSDirectConnection {
|
||||
// The direct PBS connection is authoritative for the
|
||||
// backup list, but which cluster listed a snapshot is
|
||||
// evidence only this storage view has: it attributes a
|
||||
// snapshot whose VMID exists on several clusters to the
|
||||
// cluster that made it (#1639). Keep that association
|
||||
// even though the raw entry is dropped.
|
||||
if confirmationType := pbsGuestConfirmationType(backupType); confirmationType != "" && content.VMID > 0 && content.CTime > 0 {
|
||||
pbsConfirmations = append(pbsConfirmations, models.PBSGuestConfirmation{
|
||||
BackupType: confirmationType,
|
||||
VMID: content.VMID,
|
||||
Time: content.CTime,
|
||||
})
|
||||
}
|
||||
log.Debug().
|
||||
Str("instance", instanceName).
|
||||
Str("node", node.Node).
|
||||
|
|
@ -384,6 +398,7 @@ func (m *Monitor) pollStorageBackupsWithNodes(ctx context.Context, instanceName
|
|||
|
||||
// Update state with storage backups for this instance
|
||||
m.state.UpdateStorageBackupsForInstance(instanceName, allBackups)
|
||||
m.state.UpdatePBSGuestConfirmationsForInstance(instanceName, pbsConfirmations)
|
||||
|
||||
// Best-effort ingestion into recovery store (for rollups / unified backups UX).
|
||||
guestInfo := buildProxmoxGuestInfoIndex(readState)
|
||||
|
|
@ -425,6 +440,20 @@ func (m *Monitor) pollStorageBackupsWithNodes(ctx context.Context, instanceName
|
|||
m.broadcastStateUpdate()
|
||||
}
|
||||
|
||||
// pbsGuestConfirmationType maps a PVE storage-content backup type to PBS
|
||||
// snapshot nomenclature. Host and unclassifiable entries return "" and are
|
||||
// not usable as guest attribution evidence.
|
||||
func pbsGuestConfirmationType(backupType string) string {
|
||||
switch backupType {
|
||||
case "qemu":
|
||||
return "vm"
|
||||
case "lxc":
|
||||
return "ct"
|
||||
default:
|
||||
return ""
|
||||
}
|
||||
}
|
||||
|
||||
func shouldPreserveBackups(nodeCount int, hadSuccessfulNode bool, storagesWithBackup, contentSuccess int) bool {
|
||||
if nodeCount > 0 && !hadSuccessfulNode {
|
||||
return true
|
||||
|
|
@ -1330,6 +1359,7 @@ func (m *Monitor) removeFailedPVENode(instanceName string) {
|
|||
m.state.UpdateCephClustersForInstance(instanceName, []models.CephCluster{})
|
||||
m.state.UpdateBackupTasksForInstance(instanceName, []models.BackupTask{})
|
||||
m.state.UpdateStorageBackupsForInstance(instanceName, []models.StorageBackup{})
|
||||
m.state.UpdatePBSGuestConfirmationsForInstance(instanceName, nil)
|
||||
m.state.UpdateGuestSnapshotsForInstance(instanceName, []models.GuestSnapshot{})
|
||||
|
||||
// Set connection health to false
|
||||
|
|
|
|||
|
|
@ -68,3 +68,76 @@ func TestPollStorageBackups_KeepsDirStorageNamedLikePBS(t *testing.T) {
|
|||
t.Fatalf("expected vzdump from dir storage named pbs-backup, got %+v", backups[0])
|
||||
}
|
||||
}
|
||||
|
||||
// pbsCollisionStorageClient serves one pbs-type storage listing a single
|
||||
// snapshot for VMID 173, standing in for cluster-a's own PBS storage view.
|
||||
type pbsCollisionStorageClient struct {
|
||||
mockPVEClientExtra
|
||||
snapshotTime time.Time
|
||||
}
|
||||
|
||||
func (c *pbsCollisionStorageClient) GetStorage(ctx context.Context, node string) ([]pveapi.Storage, error) {
|
||||
return []pveapi.Storage{
|
||||
{Storage: "pbs-store", Content: "backup", Type: "pbs", Enabled: 1, Active: 1},
|
||||
}, nil
|
||||
}
|
||||
|
||||
func (c *pbsCollisionStorageClient) GetStorageContent(ctx context.Context, node, storage string) ([]pveapi.StorageContent, error) {
|
||||
return []pveapi.StorageContent{{
|
||||
Volid: "pbs-store:backup/vm/173/2026-07-27T01:00:00Z",
|
||||
VMID: 173,
|
||||
Size: 2048,
|
||||
CTime: c.snapshotTime.Unix(),
|
||||
Content: "backup",
|
||||
Format: "pbs-vm",
|
||||
}}, nil
|
||||
}
|
||||
|
||||
// TestPollStorageBackups_Issue1639HarvestsPBSGuestConfirmations verifies
|
||||
// that snapshots skipped in favor of the direct PBS connection still leave
|
||||
// PVE-side attribution evidence: the cluster whose storage listed a
|
||||
// snapshot gets it as LastBackup even though the VMID exists on another
|
||||
// cluster too, and the other cluster's unconfirmed snapshot is not guessed.
|
||||
func TestPollStorageBackups_Issue1639HarvestsPBSGuestConfirmations(t *testing.T) {
|
||||
timeA := time.Date(2026, 7, 27, 1, 0, 0, 0, time.UTC)
|
||||
timeB := time.Date(2026, 7, 27, 2, 0, 0, 0, time.UTC)
|
||||
|
||||
state := models.NewState()
|
||||
state.UpdateVMs([]models.VM{
|
||||
{VMID: 173, Name: "web", Instance: "cluster-a", Node: "node-a", Status: "running"},
|
||||
{VMID: 173, Name: "web", Instance: "cluster-b", Node: "node-b", Status: "running"},
|
||||
})
|
||||
state.UpdatePBSBackups("pbs-1", []models.PBSBackup{
|
||||
{ID: "a-173", VMID: "173", BackupType: "vm", BackupTime: timeA, Instance: "pbs-1", Datastore: "backups", Owner: "shared@pbs!token"},
|
||||
{ID: "b-173", VMID: "173", BackupType: "vm", BackupTime: timeB, Instance: "pbs-1", Datastore: "backups", Owner: "shared@pbs!token"},
|
||||
})
|
||||
|
||||
m := &Monitor{
|
||||
state: state,
|
||||
config: &config.Config{
|
||||
PBSInstances: []config.PBSInstance{{Name: "pbs-1", Host: "https://pbs.example:8007"}},
|
||||
},
|
||||
}
|
||||
|
||||
client := &pbsCollisionStorageClient{snapshotTime: timeA}
|
||||
nodes := []pveapi.Node{{Node: "node-a", Status: "online"}}
|
||||
|
||||
m.pollStorageBackupsWithNodes(context.Background(), "cluster-a", client, nodes, map[string]string{"node-a": "online"})
|
||||
|
||||
snapshot := m.state.GetSnapshot()
|
||||
if len(snapshot.PVEBackups.StorageBackups) != 0 {
|
||||
t.Fatalf("pbs-type storage contents must stay out of StorageBackups, got %+v", snapshot.PVEBackups.StorageBackups)
|
||||
}
|
||||
for _, vm := range snapshot.VMs {
|
||||
switch vm.Instance {
|
||||
case "cluster-a":
|
||||
if !vm.LastBackup.Equal(timeA) {
|
||||
t.Errorf("cluster-a VM 173 LastBackup = %v, want confirmed snapshot %v", vm.LastBackup, timeA)
|
||||
}
|
||||
case "cluster-b":
|
||||
if !vm.LastBackup.IsZero() {
|
||||
t.Errorf("cluster-b VM 173 LastBackup = %v, want zero (its snapshot is unconfirmed)", vm.LastBackup)
|
||||
}
|
||||
}
|
||||
}
|
||||
}
|
||||
|
|
|
|||
|
|
@ -950,3 +950,48 @@ func TestBuildProxmoxGuestInfoIndex_UsesCanonicalReadState(t *testing.T) {
|
|||
assertInfo("pve1|nodeA|100", "vm-store-100", "pve1:nodeA:100", unifiedresources.ResourceTypeVM, "vm100")
|
||||
assertInfo("pve1|nodeB|200", "ct-store-200", "pve1:nodeB:200", unifiedresources.ResourceTypeSystemContainer, "ct200")
|
||||
}
|
||||
|
||||
// TestRetirePVEInstanceRuntimeClearsPBSGuestConfirmations verifies that
|
||||
// retiring a consolidated PVE instance also drops its PVE-side PBS
|
||||
// snapshot confirmations (#1639 evidence). Stale confirmations from a
|
||||
// retired duplicate would make every snapshot it once listed look
|
||||
// multi-confirmed and degrade collision attribution for the surviving
|
||||
// instance.
|
||||
func TestRetirePVEInstanceRuntimeClearsPBSGuestConfirmations(t *testing.T) {
|
||||
backupTime := time.Date(2026, 7, 27, 1, 0, 0, 0, time.UTC)
|
||||
|
||||
state := models.NewState()
|
||||
state.UpdateVMs([]models.VM{
|
||||
{VMID: 173, Name: "web", Instance: "cluster-a", Node: "node-a", Status: "running"},
|
||||
{VMID: 173, Name: "web", Instance: "cluster-b", Node: "node-b", Status: "running"},
|
||||
})
|
||||
state.UpdatePBSBackups("pbs-1", []models.PBSBackup{
|
||||
{ID: "a-173", VMID: "173", BackupType: "vm", BackupTime: backupTime, Instance: "pbs-1", Datastore: "backups"},
|
||||
})
|
||||
state.UpdatePBSGuestConfirmationsForInstance("cluster-a", []models.PBSGuestConfirmation{
|
||||
{BackupType: "vm", VMID: 173, Time: backupTime.Unix()},
|
||||
})
|
||||
|
||||
state.SyncGuestBackupTimes()
|
||||
for _, vm := range state.GetSnapshot().VMs {
|
||||
if vm.Instance == "cluster-a" && !vm.LastBackup.Equal(backupTime) {
|
||||
t.Fatalf("precondition: confirmed snapshot should attribute to cluster-a, got %v", vm.LastBackup)
|
||||
}
|
||||
}
|
||||
|
||||
m := &Monitor{state: state}
|
||||
m.retirePVEInstanceRuntime("cluster-a")
|
||||
|
||||
// Re-add the guest without confirmations: the retired instance's
|
||||
// evidence must be gone, so the collision VMID is unattributable again.
|
||||
state.UpdateVMs([]models.VM{
|
||||
{VMID: 173, Name: "web", Instance: "cluster-a", Node: "node-a", Status: "running"},
|
||||
{VMID: 173, Name: "web", Instance: "cluster-b", Node: "node-b", Status: "running"},
|
||||
})
|
||||
state.SyncGuestBackupTimes()
|
||||
for _, vm := range state.GetSnapshot().VMs {
|
||||
if vm.VMID == 173 && !vm.LastBackup.IsZero() {
|
||||
t.Fatalf("VM 173 on %s should be unattributable after retirement cleared confirmations, got %v", vm.Instance, vm.LastBackup)
|
||||
}
|
||||
}
|
||||
}
|
||||
|
|
|
|||
|
|
@ -159,6 +159,7 @@ func (m *Monitor) retirePVEInstanceRuntime(instanceName string) {
|
|||
m.state.UpdateCephClustersForInstance(instanceName, []models.CephCluster{})
|
||||
m.state.UpdateBackupTasksForInstance(instanceName, []models.BackupTask{})
|
||||
m.state.UpdateStorageBackupsForInstance(instanceName, []models.StorageBackup{})
|
||||
m.state.UpdatePBSGuestConfirmationsForInstance(instanceName, nil)
|
||||
m.state.UpdateGuestSnapshotsForInstance(instanceName, []models.GuestSnapshot{})
|
||||
m.state.UpdateReplicationJobsForInstance(instanceName, []models.ReplicationJob{})
|
||||
}
|
||||
|
|
|
|||
|
|
@ -88,3 +88,98 @@ func BackupGuestMatchScore(namespace, comment, vmid, guestName, instanceName, no
|
|||
}
|
||||
return score
|
||||
}
|
||||
|
||||
// LocationLabelsEqual reports whether two location labels normalize to the
|
||||
// same non-empty token. Unlike NamespaceMatchesLocation it never suffix
|
||||
// matches: callers use it when the label is a connection or instance name
|
||||
// rather than a PBS namespace segment, where loose suffix matching can
|
||||
// cross-attribute clusters that share a VMID (#1639).
|
||||
func LocationLabelsEqual(a, b string) bool {
|
||||
na := normalizeLocationLabel(a)
|
||||
nb := normalizeLocationLabel(b)
|
||||
return na != "" && na == nb
|
||||
}
|
||||
|
||||
const (
|
||||
pbsSourceOwnerKey = "owner"
|
||||
pbsSourceDatastoreKey = "datastore"
|
||||
pbsSourceInstanceKey = "instance"
|
||||
)
|
||||
|
||||
// PBSSourceLearner accumulates which PVE connection each PBS submission
|
||||
// source has been positively attributed to. A "source" is, strongest first,
|
||||
// the backup owner token, the datastore, and the PBS instance — each scoped
|
||||
// to the PBS instance the backup came from. Clusters usually push to PBS
|
||||
// with a per-cluster token and often a per-cluster datastore, so evidence
|
||||
// learned from unambiguous guests can attribute root-namespace,
|
||||
// comment-less snapshots whose VMID exists on more than one cluster (#1639).
|
||||
type PBSSourceLearner struct {
|
||||
instancesByKey map[string]map[string]struct{}
|
||||
}
|
||||
|
||||
func NewPBSSourceLearner() *PBSSourceLearner {
|
||||
return &PBSSourceLearner{instancesByKey: make(map[string]map[string]struct{})}
|
||||
}
|
||||
|
||||
func pbsSourceKeys(pbsInstance, datastore, owner string) []string {
|
||||
pbsInstance = strings.TrimSpace(pbsInstance)
|
||||
datastore = strings.TrimSpace(datastore)
|
||||
owner = strings.ToLower(strings.TrimSpace(owner))
|
||||
keys := make([]string, 0, 3)
|
||||
if owner != "" {
|
||||
keys = append(keys, pbsSourceOwnerKey+"\x00"+pbsInstance+"\x00"+owner)
|
||||
}
|
||||
if datastore != "" {
|
||||
keys = append(keys, pbsSourceDatastoreKey+"\x00"+pbsInstance+"\x00"+datastore)
|
||||
}
|
||||
if pbsInstance != "" {
|
||||
keys = append(keys, pbsSourceInstanceKey+"\x00"+pbsInstance)
|
||||
}
|
||||
return keys
|
||||
}
|
||||
|
||||
// Observe records that a backup from the given PBS source was positively
|
||||
// attributed to a PVE connection (via a unique VMID, namespace placement,
|
||||
// or guest-name match).
|
||||
func (l *PBSSourceLearner) Observe(pbsInstance, datastore, owner, pveInstance string) {
|
||||
if l == nil {
|
||||
return
|
||||
}
|
||||
pveInstance = strings.TrimSpace(pveInstance)
|
||||
if pveInstance == "" {
|
||||
return
|
||||
}
|
||||
for _, key := range pbsSourceKeys(pbsInstance, datastore, owner) {
|
||||
set, ok := l.instancesByKey[key]
|
||||
if !ok {
|
||||
set = make(map[string]struct{})
|
||||
l.instancesByKey[key] = set
|
||||
}
|
||||
set[pveInstance] = struct{}{}
|
||||
}
|
||||
}
|
||||
|
||||
// Resolve reports the single PVE connection the backup's source evidence
|
||||
// identifies. Evidence is consulted strongest-first; a source component that
|
||||
// maps to several connections is not a discriminator and defers to the next
|
||||
// one, but a component that was never observed stops resolution entirely —
|
||||
// an unfamiliar owner or datastore means the backup may belong to a cluster
|
||||
// we have no evidence for, and guessing from weaker components would
|
||||
// attribute it to the wrong cluster.
|
||||
func (l *PBSSourceLearner) Resolve(pbsInstance, datastore, owner string) (string, bool) {
|
||||
if l == nil {
|
||||
return "", false
|
||||
}
|
||||
for _, key := range pbsSourceKeys(pbsInstance, datastore, owner) {
|
||||
set, ok := l.instancesByKey[key]
|
||||
if !ok {
|
||||
return "", false
|
||||
}
|
||||
if len(set) == 1 {
|
||||
for instance := range set {
|
||||
return instance, true
|
||||
}
|
||||
}
|
||||
}
|
||||
return "", false
|
||||
}
|
||||
|
|
|
|||
|
|
@ -69,3 +69,66 @@ func TestBackupGuestMatchScoreRanksNodeMatchAboveClusterEntrypoint(t *testing.T)
|
|||
t.Fatalf("node namespace score = %d, should outrank weak instance+comment match", got)
|
||||
}
|
||||
}
|
||||
|
||||
func TestLocationLabelsEqualRequiresExactNormalizedMatch(t *testing.T) {
|
||||
if !LocationLabelsEqual("pve-nat", "PVE Nat") {
|
||||
t.Fatal("normalized identical labels should match")
|
||||
}
|
||||
// NamespaceMatchesLocation would suffix-match these; a connection label
|
||||
// must not (#1639).
|
||||
if LocationLabelsEqual("nat", "pve-nat") {
|
||||
t.Fatal("suffix relation between distinct labels must not match")
|
||||
}
|
||||
if LocationLabelsEqual("", "") {
|
||||
t.Fatal("empty labels must not match")
|
||||
}
|
||||
}
|
||||
|
||||
func TestPBSSourceLearnerResolvesOwnerBeforeWeakerComponents(t *testing.T) {
|
||||
l := NewPBSSourceLearner()
|
||||
l.Observe("pbs-main", "backups", "cluster-a@pbs!token", "cluster-a")
|
||||
l.Observe("pbs-main", "backups", "cluster-b@pbs!token", "cluster-b")
|
||||
|
||||
// Datastore and PBS instance are shared, but each owner token uniquely
|
||||
// identifies its cluster.
|
||||
if inst, ok := l.Resolve("pbs-main", "backups", "cluster-a@pbs!token"); !ok || inst != "cluster-a" {
|
||||
t.Fatalf("owner resolution = %q,%v, want cluster-a,true", inst, ok)
|
||||
}
|
||||
if inst, ok := l.Resolve("pbs-main", "backups", "CLUSTER-B@pbs!token"); !ok || inst != "cluster-b" {
|
||||
t.Fatalf("owner resolution should be case-insensitive, got %q,%v", inst, ok)
|
||||
}
|
||||
}
|
||||
|
||||
func TestPBSSourceLearnerSharedSourceIsNotDecisive(t *testing.T) {
|
||||
l := NewPBSSourceLearner()
|
||||
l.Observe("pbs-main", "backups", "shared@pbs!token", "cluster-a")
|
||||
l.Observe("pbs-main", "backups", "shared@pbs!token", "cluster-b")
|
||||
|
||||
if inst, ok := l.Resolve("pbs-main", "backups", "shared@pbs!token"); ok {
|
||||
t.Fatalf("shared source resolved to %q, want inconclusive", inst)
|
||||
}
|
||||
}
|
||||
|
||||
func TestPBSSourceLearnerUnfamiliarComponentStopsResolution(t *testing.T) {
|
||||
l := NewPBSSourceLearner()
|
||||
l.Observe("pbs-main", "backups", "cluster-a@pbs!token", "cluster-a")
|
||||
|
||||
// The datastore alone would resolve to cluster-a, but an owner token
|
||||
// that was never observed means the backup may come from a cluster we
|
||||
// have no evidence for — weaker components must not be consulted.
|
||||
if inst, ok := l.Resolve("pbs-main", "backups", "mystery@pbs!token"); ok {
|
||||
t.Fatalf("unfamiliar owner resolved to %q, want inconclusive", inst)
|
||||
}
|
||||
// Without an owner on the backup, the datastore evidence applies.
|
||||
if inst, ok := l.Resolve("pbs-main", "backups", ""); !ok || inst != "cluster-a" {
|
||||
t.Fatalf("datastore resolution = %q,%v, want cluster-a,true", inst, ok)
|
||||
}
|
||||
}
|
||||
|
||||
func TestPBSSourceLearnerNilReceiverIsInert(t *testing.T) {
|
||||
var l *PBSSourceLearner
|
||||
l.Observe("pbs-main", "backups", "owner", "cluster-a")
|
||||
if _, ok := l.Resolve("pbs-main", "backups", "owner"); ok {
|
||||
t.Fatal("nil learner must never be decisive")
|
||||
}
|
||||
}
|
||||
|
|
|
|||
|
|
@ -53,7 +53,7 @@ func preferredPBSBackupSubjectName(comment, vmid string) string {
|
|||
return proxmoxidentity.PreferredPBSBackupSubjectName(comment, vmid)
|
||||
}
|
||||
|
||||
func selectPBSGuestCandidate(backup models.PBSBackup, candidates []GuestCandidate) (GuestCandidate, bool) {
|
||||
func selectPBSGuestCandidate(backup models.PBSBackup, candidates []GuestCandidate, learner *proxmoxidentity.PBSSourceLearner) (GuestCandidate, bool) {
|
||||
if len(candidates) == 0 {
|
||||
return GuestCandidate{}, false
|
||||
}
|
||||
|
|
@ -92,6 +92,30 @@ func selectPBSGuestCandidate(backup models.PBSBackup, candidates []GuestCandidat
|
|||
}
|
||||
}
|
||||
|
||||
// Last resort for collision VMIDs with no namespace or name evidence:
|
||||
// the learned submission-source mapping (owner token / datastore / PBS
|
||||
// instance, learned from this batch's attributable backups) can name the
|
||||
// cluster the backup came from (#1639). A decisive attribution to a
|
||||
// connection with no candidate here means the backup's own guest is
|
||||
// gone; never link it to another cluster's guest.
|
||||
if learner != nil {
|
||||
if attributed, decisive := learner.Resolve(backup.Instance, backup.Datastore, backup.Owner); decisive {
|
||||
filtered := make([]GuestCandidate, 0, len(matched))
|
||||
for _, candidate := range matched {
|
||||
if candidate.InstanceName == attributed {
|
||||
filtered = append(filtered, candidate)
|
||||
}
|
||||
}
|
||||
if len(filtered) == 1 {
|
||||
return filtered[0], true
|
||||
}
|
||||
if len(filtered) == 0 {
|
||||
return GuestCandidate{}, false
|
||||
}
|
||||
matched = filtered
|
||||
}
|
||||
}
|
||||
|
||||
if len(matched) == 1 {
|
||||
return matched[0], true
|
||||
}
|
||||
|
|
@ -380,6 +404,20 @@ func FromPBSBackups(backups []models.PBSBackup, candidatesByKey map[string][]Gue
|
|||
return nil
|
||||
}
|
||||
|
||||
// First pass: learn each PBS submission source's cluster from the
|
||||
// backups that are attributable on their own evidence, so the second
|
||||
// pass can resolve collision VMIDs with no evidence of their own (#1639).
|
||||
learner := proxmoxidentity.NewPBSSourceLearner()
|
||||
for _, b := range backups {
|
||||
if strings.TrimSpace(b.ID) == "" {
|
||||
continue
|
||||
}
|
||||
key := strings.ToLower(strings.TrimSpace(b.BackupType)) + ":" + strings.TrimSpace(b.VMID)
|
||||
if c, ok := selectPBSGuestCandidate(b, candidatesByKey[key], nil); ok {
|
||||
learner.Observe(b.Instance, b.Datastore, b.Owner, c.InstanceName)
|
||||
}
|
||||
}
|
||||
|
||||
out := make([]recovery.RecoveryPoint, 0, len(backups))
|
||||
for _, b := range backups {
|
||||
if strings.TrimSpace(b.ID) == "" {
|
||||
|
|
@ -395,7 +433,7 @@ func FromPBSBackups(backups []models.PBSBackup, candidatesByKey map[string][]Gue
|
|||
|
||||
// Link to a unified resource when the candidate set is already singular or can be
|
||||
// disambiguated by PBS namespace / guest label without guessing across guest collisions.
|
||||
if c, ok := selectPBSGuestCandidate(b, candidates); ok {
|
||||
if c, ok := selectPBSGuestCandidate(b, candidates, learner); ok {
|
||||
subjectRID = subjectResourceID(c.ResourceType, c.ResourceID, c.SourceID)
|
||||
subjectRef = proxmoxSubjectRef(c.ResourceType, GuestInfo{Name: c.DisplayName, ResourceType: c.ResourceType, SourceID: c.SourceID}, c.InstanceName, c.NodeName, c.VMID, c.SourceID)
|
||||
} else {
|
||||
|
|
|
|||
|
|
@ -510,3 +510,100 @@ func TestFromPBSBackupsWithEvidenceAddsProviderScopeAndCorrelation(t *testing.T)
|
|||
t.Fatalf("evidence Validate() error = %v", err)
|
||||
}
|
||||
}
|
||||
|
||||
// Issue #1639: a root-namespace comment-less snapshot whose VMID exists on
|
||||
// two clusters must link to the cluster its submission source (owner token,
|
||||
// datastore, PBS instance) belongs to, learned from the batch's
|
||||
// attributable backups — and must stay unlinked when the source is shared.
|
||||
func TestFromPBSBackups_DisambiguatesCollisionVMIDBySubmissionSource(t *testing.T) {
|
||||
candidatesByKey := map[string][]GuestCandidate{
|
||||
"vm:173": {
|
||||
{
|
||||
ResourceID: "vm-aaaaaaaaaaaaaaaa",
|
||||
SourceID: "cluster-a:pve-a1:173",
|
||||
ResourceType: unifiedresources.ResourceTypeVM,
|
||||
DisplayName: "web-a",
|
||||
InstanceName: "cluster-a",
|
||||
NodeName: "pve-a1",
|
||||
VMID: 173,
|
||||
BackupTypeKey: "vm",
|
||||
},
|
||||
{
|
||||
ResourceID: "vm-bbbbbbbbbbbbbbbb",
|
||||
SourceID: "cluster-b:pve-b1:173",
|
||||
ResourceType: unifiedresources.ResourceTypeVM,
|
||||
DisplayName: "web-b",
|
||||
InstanceName: "cluster-b",
|
||||
NodeName: "pve-b1",
|
||||
VMID: 173,
|
||||
BackupTypeKey: "vm",
|
||||
},
|
||||
},
|
||||
"vm:100": {
|
||||
{
|
||||
ResourceID: "vm-cccccccccccccccc",
|
||||
SourceID: "cluster-a:pve-a1:100",
|
||||
ResourceType: unifiedresources.ResourceTypeVM,
|
||||
DisplayName: "db-a",
|
||||
InstanceName: "cluster-a",
|
||||
NodeName: "pve-a1",
|
||||
VMID: 100,
|
||||
BackupTypeKey: "vm",
|
||||
},
|
||||
},
|
||||
}
|
||||
|
||||
backups := []models.PBSBackup{
|
||||
// Teaching snapshot: unique VMID attributes cluster-a's owner token.
|
||||
{
|
||||
ID: "pbs-a-100",
|
||||
VMID: "100",
|
||||
Instance: "pbs-main",
|
||||
Datastore: "store-a",
|
||||
Owner: "cluster-a@pbs!token",
|
||||
BackupType: "vm",
|
||||
BackupTime: time.Date(2026, 7, 26, 1, 0, 0, 0, time.UTC),
|
||||
},
|
||||
// Collision snapshot: no namespace, no comment, cluster-a's source.
|
||||
{
|
||||
ID: "pbs-a-173",
|
||||
VMID: "173",
|
||||
Instance: "pbs-main",
|
||||
Datastore: "store-a",
|
||||
Owner: "cluster-a@pbs!token",
|
||||
BackupType: "vm",
|
||||
BackupTime: time.Date(2026, 7, 27, 1, 0, 0, 0, time.UTC),
|
||||
},
|
||||
// Collision snapshot from a source never attributed to any cluster:
|
||||
// must stay unlinked rather than guess.
|
||||
{
|
||||
ID: "pbs-x-173",
|
||||
VMID: "173",
|
||||
Instance: "pbs-main",
|
||||
Datastore: "store-x",
|
||||
Owner: "mystery@pbs!token",
|
||||
BackupType: "vm",
|
||||
BackupTime: time.Date(2026, 7, 27, 2, 0, 0, 0, time.UTC),
|
||||
},
|
||||
}
|
||||
|
||||
result := FromPBSBackups(backups, candidatesByKey)
|
||||
if len(result) != 3 {
|
||||
t.Fatalf("expected 3 points, got %d", len(result))
|
||||
}
|
||||
|
||||
byID := map[string]int{}
|
||||
for i, p := range result {
|
||||
byID[p.ID] = i
|
||||
}
|
||||
|
||||
linked := result[byID["pbs-backup:pbs-a-173"]]
|
||||
if linked.SubjectResourceID != "vm-aaaaaaaaaaaaaaaa" {
|
||||
t.Fatalf("collision snapshot SubjectResourceID = %q, want cluster-a guest", linked.SubjectResourceID)
|
||||
}
|
||||
|
||||
unlinked := result[byID["pbs-backup:pbs-x-173"]]
|
||||
if unlinked.SubjectResourceID != "" {
|
||||
t.Fatalf("unattributable snapshot linked to %q, want unlinked", unlinked.SubjectResourceID)
|
||||
}
|
||||
}
|
||||
|
|
|
|||
Loading…
Add table
Add a link
Reference in a new issue