Harden provider MSP tenant isolation

- isolate provider MSP tenant runtimes onto per-client Docker networks - add adversarial proofs for workspace cap races, handoff retargeting, and org-bound agent report retargeting - pin provider deploy proof to start Traefik before tenant network creation
2026-07-09 16:00:59 +00:00 · 2026-06-02 17:34:10 +01:00 · 2026-06-02 17:34:10 +01:00 · d729461cad
commit d729461cad
parent 473a0ddee6
15 changed files with 697 additions and 18 deletions
--- a/cmd/pulse-control-plane/provider_msp_preflight.go
+++ b/cmd/pulse-control-plane/provider_msp_preflight.go
@ -186,6 +186,7 @@ func providerMSPDockerManagerConfig(cfg *cloudcp.CPConfig) cpDocker.ManagerConfi
 	return cpDocker.ManagerConfig{
 		Image:                    cfg.PulseImage,
 		Network:                  cfg.DockerNetwork,
+		IsolateTenantNetworks:    true,
 		BaseDomain:               providerMSPBaseDomainFromURL(cfg.BaseURL),
 		TrialActivationPublicKey: cfg.TrialActivationPublicKey,
 		TrustedProxyCIDRs:        cfg.TrustedProxyCIDRs,
--- a/deploy/provider-msp/docker-compose.yml
+++ b/deploy/provider-msp/docker-compose.yml
@ -18,6 +18,8 @@ services:
    networks:
      - pulse-provider-msp
    restart: unless-stopped
+    labels:
+      - pulse.provider-msp.role=traefik

  control-plane:
    image: ${CONTROL_PLANE_IMAGE:?CONTROL_PLANE_IMAGE must be set to a digest-pinned image}
@ -64,6 +66,7 @@ services:
      - traefik
    restart: unless-stopped
    labels:
+      - pulse.provider-msp.role=control-plane
      - traefik.enable=true
      - traefik.docker.network=${PULSE_PROVIDER_MSP_DOCKER_NETWORK:-pulse-provider-msp}
      - traefik.http.routers.provider-control-plane.rule=Host(`${DOMAIN}`)
--- a/deploy/provider-msp/run-install-proof.sh
+++ b/deploy/provider-msp/run-install-proof.sh
@ -9,9 +9,10 @@ Usage:
 Runs the provider-hosted MSP compose install proof on a Docker host:
  1. validates .env and docker-compose.yml
  2. optionally pulls the pinned Traefik and control-plane images
-  3. runs provider-msp install-proof through the compose control-plane service
-  4. starts the provider stack
-  5. runs provider-msp status as a final operator check
+  3. starts Traefik so proof workspaces can attach isolated tenant networks
+  4. runs provider-msp install-proof through the compose control-plane service
+  5. starts the provider stack
+  6. runs provider-msp status as a final operator check

 Options:
  --account-name NAME          Provider MSP account display name
@ -121,6 +122,8 @@ if [[ "$skip_compose_pull" != "1" ]]; then
  docker compose pull traefik control-plane
 fi

+docker compose up -d traefik
+
 install_args=(
  provider-msp install-proof
  --account-name "$account_name"
--- a/docs/release-control/v6/internal/subsystems/cloud-paid.md
+++ b/docs/release-control/v6/internal/subsystems/cloud-paid.md
@ -37,6 +37,11 @@ Provider-hosted MSP backup, verification, and restore are part of the cloud-paid
 operator contract because the recovery artifact must preserve the provider's
 license-backed plan identity, control-plane account/workspace registry, and
 tenant-local runtime state without depending on Stripe billing surfaces.
+Provider-hosted MSP tenant runtime isolation is part of the cloud-paid runtime
+contract. `CP_DOCKER_NETWORK` names the provider ingress and support network,
+not the client runtime boundary: each client workspace runtime must be created
+on a derived per-tenant Docker bridge, and support containers must be explicitly
+attached to that bridge before the tenant runtime is started.

 ## Canonical Files

@ -192,6 +197,18 @@ tenant-local runtime state without depending on Stripe billing surfaces.
   Pulse Pro operations bundle on a recurring systemd timer, write durable
   status/log output, and emit Prometheus textfile metrics so a clean GA
   baseline is continuously monitored rather than manually rediscovered.
+   Provider-hosted MSP runtime isolation must keep client tenant runtimes off
+   the shared provider ingress/support network. The Docker manager must derive
+   a per-client tenant bridge, label it with tenant identity, pin Traefik
+   routing to that tenant bridge, attach only provider support containers
+   selected by provider MSP labels, start the tenant runtime only after those
+   support attachments succeed, prefer the tenant bridge for health checks, and
+   remove tenant bridges when the owning runtime is removed.
+   Client-bound proof must cover the boundary itself: workspace limits must not
+   be raceable past the licensed cap, handoff tokens must not be replayed or
+   retargeted across workspaces, org-bound agent install/report tokens must not
+   write into another client runtime, and rotated-out install tokens must fail
+   immediately.
 10. `internal/cloudcp/provider_msp_backup.go` shared with `deployment-installability`: provider-hosted MSP backup is both a cloud-paid license/account/runtime continuity boundary and a deployment-installability recovery artifact boundary.
    License-backed provider MSP backups must include the signed MSP license
    file as a recovery artifact while exposing only license metadata in command
@ -212,6 +229,10 @@ tenant-local runtime state without depending on Stripe billing surfaces.
    coherent.
    Tenant runtime rollout and missing-runtime restore must fail closed on the
    same storage admission guard before snapshotting or swapping containers.
+    Provider-hosted MSP rollout and recovery must preserve the same isolated
+    tenant network contract when recreating, reconciling, or rolling tenant
+    containers; a recovered client workspace must not fall back to the shared
+    provider ingress/support network as its runtime network.
    `tenant-runtime rollout --all --image <target>` is the canonical hosted
    fleet image upgrade path for active tenant runtimes. Its `--dry-run` mode
    must print the target-image rollout plan before mutation, `--all` must
--- a/docs/release-control/v6/internal/subsystems/deployment-installability.md
+++ b/docs/release-control/v6/internal/subsystems/deployment-installability.md
@ -129,6 +129,13 @@ surfaces.
   the configured tenant runtime image, configured Docker network, Docker
   daemon reachability, and storage-admission guardrails before the provider
   treats a fresh install as ready for client onboarding.
+   Provider-hosted MSP installability treats `CP_DOCKER_NETWORK` as the
+   provider ingress/support network, not the client runtime network. Each
+   workspace runtime must be created on a per-tenant bridge derived by the
+   Docker manager, with Traefik routing pinned to that bridge through
+   `traefik.docker.network`. The packaged compose stack must label the Traefik
+   and control-plane support containers so the control plane can attach them to
+   each tenant bridge before starting the client runtime.
   `pulse-control-plane provider-msp proof` must exercise the first-client
   onboarding path through workspace creation, client-bound install token
   generation, tenant-local unified-agent report ingest, tenant-bound install
@ -137,6 +144,11 @@ surfaces.
   treated as proven. The proof is license-backed by default: `license_file` must be the
   resolved provider MSP plan source unless the operator explicitly opts into
   the local-development `--allow-env-plan` escape hatch.
+   The same proof surface must also keep adversarial client-boundary probes in
+   scope: workspace-limit check/create must be locked against concurrent cap
+   bypass, handoff tokens must reject cross-workspace retargeting without being
+   consumed, org-bound agent report tokens must not write into another client
+   runtime, and rotated-out install tokens must be rejected immediately.
   `pulse-control-plane provider-msp install-proof` is the packaged fresh
   install rehearsal: it must bootstrap the provider owner, run license-backed
   preflight and status checks, run the workspace/runtime proof with cleanup
@ -147,9 +159,11 @@ surfaces.
   `deploy/provider-msp/run-install-proof.sh` is the compose-level operator
   wrapper for that rehearsal. It must validate the provider `.env` and compose
   config, require a reachable Docker daemon, optionally pull the pinned
-   provider images, run the one-off `provider-msp install-proof` command through
-   the packaged control-plane service, start the long-running provider stack,
-   and finish with `provider-msp status`.
+   provider images, start Traefik before proof workspace creation so isolated
+   tenant bridges can attach their ingress support container, run the one-off
+   `provider-msp install-proof` command through the packaged control-plane
+   service, start the long-running provider stack, and finish with
+   `provider-msp status`.
   `deploy/provider-msp/upgrade.sh` is the compose-level pre-upgrade and
   pre-maintenance runner for provider-hosted MSP. It must keep dry-run mode
   non-mutating, validate the provider `.env` and compose config, check
--- a/internal/api/cloud_handoff_handlers_test.go
+++ b/internal/api/cloud_handoff_handlers_test.go
@ -464,6 +464,61 @@ func TestHandleHandoffExchange(t *testing.T) {
 	})
 }

+func TestHandleHandoffExchangeRejectsRetargetWithoutConsumingToken(t *testing.T) {
+	key := []byte("test-handoff-key")
+	configDir := t.TempDir()
+	resetSessionStoreForTests()
+	t.Cleanup(resetSessionStoreForTests)
+	resetCSRFStoreForTests()
+	t.Cleanup(resetCSRFStoreForTests)
+	InitSessionStore(configDir)
+	InitCSRFStore(configDir)
+	secretsDir := filepath.Join(configDir, "secrets")
+	if err := os.MkdirAll(secretsDir, 0o755); err != nil {
+		t.Fatalf("MkdirAll() error = %v", err)
+	}
+	if err := os.WriteFile(filepath.Join(secretsDir, "handoff.key"), key, 0o600); err != nil {
+		t.Fatalf("WriteFile() error = %v", err)
+	}
+
+	for _, tenantID := range []string{"tenant-a", "tenant-b"} {
+		saveHandoffTestOrganization(t, configDir, &models.Organization{
+			ID:          tenantID,
+			DisplayName: tenantID,
+			Status:      models.OrgStatusActive,
+			CreatedAt:   time.Now().UTC(),
+			OwnerUserID: "user-retarget",
+			Members: []models.OrganizationMember{
+				{UserID: "user-retarget", Email: "retarget@example.com", Role: models.OrgRoleOwner, AddedAt: time.Now().UTC()},
+			},
+		})
+	}
+
+	token := signHandoffToken(t, key, cloudHandoffClaims{
+		AccountID: "acct-retarget",
+		Email:     "retarget@example.com",
+		Role:      "owner",
+		RegisteredClaims: jwt.RegisteredClaims{
+			ID:        "jti-retarget",
+			Subject:   "user-retarget",
+			Issuer:    cloudHandoffIssuer,
+			Audience:  jwt.ClaimStrings{"tenant-a"},
+			ExpiresAt: jwt.NewNumericDate(time.Now().Add(time.Hour)),
+		},
+	})
+	handler := HandleHandoffExchange(configDir)
+
+	retargeted := makeExchangeRequest(t, handler, "tenant-b.example.com", token)
+	if retargeted.Code != http.StatusUnauthorized {
+		t.Fatalf("retargeted status = %d, want %d: %s", retargeted.Code, http.StatusUnauthorized, retargeted.Body.String())
+	}
+
+	original := makeExchangeRequest(t, handler, "tenant-a.example.com", token)
+	if original.Code != http.StatusOK {
+		t.Fatalf("original tenant status = %d, want %d after failed retarget: %s", original.Code, http.StatusOK, original.Body.String())
+	}
+}
+
 func TestHandleHandoffExchangeBrowserFlowSetsSessionCookies(t *testing.T) {
 	key := []byte("test-handoff-key")
 	configDir := t.TempDir()
--- a/internal/api/hosted_agent_install_command_test.go
+++ b/internal/api/hosted_agent_install_command_test.go
@ -1,15 +1,18 @@
 package api

 import (
+	"bytes"
 	"encoding/json"
 	"net/http"
 	"net/http/httptest"
 	"regexp"
 	"strings"
 	"testing"
+	"time"

 	"github.com/rcourtman/pulse-go-rewrite/internal/config"
 	"github.com/rcourtman/pulse-go-rewrite/internal/monitoring"
+	agentshost "github.com/rcourtman/pulse-go-rewrite/pkg/agents/host"
 	"github.com/stretchr/testify/require"
 )

@ -154,3 +157,80 @@ func TestGenerateHostedTenantAgentInstallCommandEnforcesHostedOrgBoundary(t *tes
 	_, ok := (&config.Config{APITokens: tokens}).ValidateAPIToken(result.Token)
 	require.True(t, ok)
 }
+
+func TestHostedTenantAgentInstallTokenCannotReportToOtherTenant(t *testing.T) {
+	defer SetMultiTenantEnabled(false)
+	SetMultiTenantEnabled(true)
+	t.Setenv("PULSE_DEV", "true")
+	setMockModeForTest(t, true)
+
+	dataDir := t.TempDir()
+	cfg := &config.Config{
+		DataPath:   dataDir,
+		ConfigPath: dataDir,
+		PublicURL:  "https://msp.example.com",
+	}
+	persistence := config.NewConfigPersistence(dataDir)
+	multiTenant := config.NewMultiTenantPersistence(dataDir)
+	for _, orgID := range []string{"client-a", "client-b"} {
+		_, err := multiTenant.GetPersistence(orgID)
+		require.NoError(t, err)
+	}
+
+	mtm := monitoring.NewMultiTenantMonitor(cfg, multiTenant, nil)
+	t.Cleanup(mtm.Stop)
+	_, err := mtm.GetMonitor("client-a")
+	require.NoError(t, err)
+
+	install, err := GenerateHostedTenantAgentInstallCommand(HostedTenantAgentInstallCommandOptions{
+		Config:        cfg,
+		Persistence:   persistence,
+		MultiTenant:   multiTenant,
+		TenantMonitor: mtm,
+		HostedMode:    true,
+		OrgID:         "client-a",
+		InstallType:   "pve",
+		OwnerUserID:   "owner-1",
+		BaseURL:       "https://client-a.msp.example.com",
+	})
+	require.NoError(t, err)
+
+	router := NewRouter(cfg, nil, mtm, nil, nil, "1.0.0")
+	report := agentshost.Report{
+		Agent: agentshost.AgentInfo{
+			ID:      "agent-client-a",
+			Version: "1.0.0",
+			Type:    "unified",
+		},
+		Host: agentshost.HostInfo{
+			ID:        "machine-client-a",
+			MachineID: "machine-client-a",
+			Hostname:  "pve1",
+			Platform:  "linux",
+		},
+		Timestamp: time.Now().UTC(),
+	}
+	body, err := json.Marshal(report)
+	require.NoError(t, err)
+
+	allowedReq := httptest.NewRequest(http.MethodPost, "/api/agents/agent/report", bytes.NewReader(body))
+	allowedReq.Header.Set("X-API-Token", install.Token)
+	allowedRec := httptest.NewRecorder()
+	router.Handler().ServeHTTP(allowedRec, allowedReq)
+	require.Equal(t, http.StatusOK, allowedRec.Code, allowedRec.Body.String())
+
+	retargetReq := httptest.NewRequest(http.MethodPost, "/api/agents/agent/report", bytes.NewReader(body))
+	retargetReq.Header.Set("X-API-Token", install.Token)
+	retargetReq.Header.Set("X-Pulse-Org-ID", "client-b")
+	retargetRec := httptest.NewRecorder()
+	router.Handler().ServeHTTP(retargetRec, retargetReq)
+	require.Equal(t, http.StatusForbidden, retargetRec.Code, retargetRec.Body.String())
+
+	monitorA, ok := mtm.PeekMonitor("client-a")
+	require.True(t, ok)
+	require.Len(t, monitorA.GetLiveHostsSnapshot(), 1)
+
+	monitorB, ok := mtm.PeekMonitor("client-b")
+	require.True(t, ok)
+	require.Empty(t, monitorB.GetLiveHostsSnapshot(), "retargeted client-a token must not write into client-b")
+}
--- a/internal/cloudcp/account/tenant_handlers_test.go
+++ b/internal/cloudcp/account/tenant_handlers_test.go
@ -12,6 +12,7 @@ import (
 	"net/http/httptest"
 	"os"
 	"path/filepath"
+	"sync"
 	"testing"

 	"github.com/rcourtman/pulse-go-rewrite/internal/cloudcp/admin"
@ -459,6 +460,67 @@ func TestCreateWorkspace_MSPStarterLimitEnforced(t *testing.T) {
 	}
 }

+func TestCreateWorkspace_MSPStarterLimitCannotRacePastHandlerLock(t *testing.T) {
+	reg := newTestRegistry(t)
+	tenantsDir := t.TempDir()
+	mux, _ := newTestTenantMux(t, reg, tenantsDir)
+
+	accountID, err := registry.GenerateAccountID()
+	if err != nil {
+		t.Fatal(err)
+	}
+	if err := reg.CreateAccount(&registry.Account{ID: accountID, Kind: registry.AccountKindMSP, DisplayName: "Race MSP"}); err != nil {
+		t.Fatal(err)
+	}
+	createTestStripeAccount(t, reg, accountID, "msp_starter")
+	workspaceLimit, known := pkglicensing.WorkspaceLimitForPlan("msp_starter")
+	if !known {
+		t.Fatal("msp_starter workspace limit is not registered")
+	}
+
+	attempts := workspaceLimit + 8
+	var wg sync.WaitGroup
+	statuses := make(chan int, attempts)
+	for i := 0; i < attempts; i++ {
+		wg.Add(1)
+		go func(i int) {
+			defer wg.Done()
+			body := fmt.Sprintf(`{"display_name":"Client %d"}`, i+1)
+			req := httptest.NewRequest(http.MethodPost, "/api/accounts/"+accountID+"/tenants", bytes.NewBufferString(body))
+			rec := doRequest(t, mux, req)
+			statuses <- rec.Code
+		}(i)
+	}
+	wg.Wait()
+	close(statuses)
+
+	created := 0
+	blocked := 0
+	for status := range statuses {
+		switch status {
+		case http.StatusCreated:
+			created++
+		case http.StatusForbidden:
+			blocked++
+		default:
+			t.Fatalf("unexpected status %d in concurrent create race", status)
+		}
+	}
+	if created != workspaceLimit {
+		t.Fatalf("created = %d, want exactly workspace limit %d", created, workspaceLimit)
+	}
+	if blocked != attempts-workspaceLimit {
+		t.Fatalf("blocked = %d, want %d", blocked, attempts-workspaceLimit)
+	}
+	tenants, err := reg.ListByAccountID(accountID)
+	if err != nil {
+		t.Fatalf("ListByAccountID: %v", err)
+	}
+	if len(tenants) != workspaceLimit {
+		t.Fatalf("tenant count = %d, want %d", len(tenants), workspaceLimit)
+	}
+}
+
 func TestCreateWorkspace_DeletedWorkspacesDoNotCount(t *testing.T) {
 	reg := newTestRegistry(t)
 	tenantsDir := t.TempDir()
--- a/internal/cloudcp/docker/labels.go
+++ b/internal/cloudcp/docker/labels.go
@ -30,11 +30,11 @@ func CanonicalTenantRuntimeRouting(tenantID, baseDomain string) TenantRuntimeRou

 // TraefikLabels generates Docker labels for Traefik reverse-proxy routing.
 // Each tenant gets a subdomain: <tenantID>.cloud.pulserelay.pro
-func TraefikLabels(tenantID, baseDomain string, containerPort int) map[string]string {
+func TraefikLabels(tenantID, baseDomain string, containerPort int, dockerNetwork ...string) map[string]string {
 	svc := "pulse-" + tenantID
 	routing := CanonicalTenantRuntimeRouting(tenantID, baseDomain)

-	return map[string]string{
+	labels := map[string]string{
 		"traefik.enable": "true",

 		// HTTP router
@ -48,4 +48,12 @@ func TraefikLabels(tenantID, baseDomain string, containerPort int) map[string]st
 		// Metadata
 		"pulse.tenant.id": tenantID,
 	}
+	for _, networkName := range dockerNetwork {
+		networkName = strings.TrimSpace(networkName)
+		if networkName != "" {
+			labels["traefik.docker.network"] = networkName
+			break
+		}
+	}
+	return labels
 }
--- a/internal/cloudcp/docker/manager.go
+++ b/internal/cloudcp/docker/manager.go
@ -24,6 +24,9 @@ import (
 type ManagerConfig struct {
 	Image                    string
 	Network                  string
+	IsolateTenantNetworks    bool
+	TenantNetworkPrefix      string
+	SupportContainerLabels   []string
 	BaseDomain               string
 	TrialActivationPublicKey string
 	TrustedProxyCIDRs        []string
@ -104,6 +107,13 @@ const (
 	defaultTenantLogMaxFile = 3
 )

+const (
+	providerSupportTraefikLabel      = "pulse.provider-msp.role=traefik"
+	providerSupportControlPlaneLabel = "pulse.provider-msp.role=control-plane"
+	tenantRuntimeNetworkLabel        = "pulse.provider-msp.network"
+	tenantRuntimeNetworkLabelValue   = "tenant"
+)
+
 // NewManager creates a Docker manager connected to the local daemon.
 func NewManager(cfg ManagerConfig) (*Manager, error) {
 	cli, err := client.New(client.FromEnv)
@ -113,6 +123,23 @@ func NewManager(cfg ManagerConfig) (*Manager, error) {
 	if cfg.ContainerPort == 0 {
 		cfg.ContainerPort = 7655
 	}
+	cfg.Network = strings.TrimSpace(cfg.Network)
+	cfg.TenantNetworkPrefix = strings.TrimSpace(cfg.TenantNetworkPrefix)
+	if cfg.IsolateTenantNetworks {
+		if cfg.TenantNetworkPrefix == "" {
+			if cfg.Network != "" {
+				cfg.TenantNetworkPrefix = cfg.Network + "-tenant"
+			} else {
+				cfg.TenantNetworkPrefix = "pulse-tenant"
+			}
+		}
+		if len(cfg.SupportContainerLabels) == 0 {
+			cfg.SupportContainerLabels = []string{
+				providerSupportTraefikLabel,
+				providerSupportControlPlaneLabel,
+			}
+		}
+	}
 	return &Manager{cli: cli, cfg: cfg}, nil
 }

@ -263,6 +290,141 @@ func (m *Manager) CheckRuntimePrerequisites(ctx context.Context, opts RuntimePre
 	return report, nil
 }

+func (m *Manager) tenantNetworkName(tenantID string) string {
+	if m == nil {
+		return ""
+	}
+	prefix := strings.TrimSpace(m.cfg.TenantNetworkPrefix)
+	if prefix == "" {
+		prefix = strings.TrimSpace(m.cfg.Network)
+	}
+	if prefix == "" {
+		prefix = "pulse-tenant"
+	}
+	return sanitizeDockerNameFragment(prefix + "-" + strings.TrimSpace(tenantID))
+}
+
+func sanitizeDockerNameFragment(raw string) string {
+	raw = strings.ToLower(strings.TrimSpace(raw))
+	if raw == "" {
+		return "pulse-tenant"
+	}
+	var b strings.Builder
+	lastDash := false
+	for _, r := range raw {
+		allowed := (r >= 'a' && r <= 'z') || (r >= '0' && r <= '9') || r == '_' || r == '.' || r == '-'
+		if !allowed {
+			if !lastDash {
+				b.WriteByte('-')
+				lastDash = true
+			}
+			continue
+		}
+		b.WriteRune(r)
+		lastDash = r == '-'
+	}
+	out := strings.Trim(b.String(), "-_.")
+	if out == "" {
+		return "pulse-tenant"
+	}
+	return out
+}
+
+func (m *Manager) ensureTenantNetwork(ctx context.Context, tenantID string) (string, error) {
+	if m == nil {
+		return "", fmt.Errorf("docker manager unavailable")
+	}
+	if !m.cfg.IsolateTenantNetworks {
+		networkName := strings.TrimSpace(m.cfg.Network)
+		if networkName == "" {
+			return "", fmt.Errorf("tenant Docker network is required")
+		}
+		return networkName, nil
+	}
+
+	networkName := m.tenantNetworkName(tenantID)
+	inspect, err := m.cli.NetworkInspect(ctx, networkName, client.NetworkInspectOptions{})
+	if err == nil {
+		if got := strings.TrimSpace(inspect.Network.Labels["pulse.tenant.id"]); got != "" && got != tenantID {
+			return "", fmt.Errorf("tenant network %q belongs to tenant %q, not %q", networkName, got, tenantID)
+		}
+		return networkName, nil
+	}
+	if !errdefs.IsNotFound(err) {
+		return "", fmt.Errorf("inspect tenant network %q: %w", networkName, err)
+	}
+
+	_, err = m.cli.NetworkCreate(ctx, networkName, client.NetworkCreateOptions{
+		Driver: "bridge",
+		Labels: map[string]string{
+			"pulse.managed":           "true",
+			"pulse.tenant.id":         tenantID,
+			tenantRuntimeNetworkLabel: tenantRuntimeNetworkLabelValue,
+		},
+	})
+	if err != nil {
+		return "", fmt.Errorf("create tenant network %q: %w", networkName, err)
+	}
+	return networkName, nil
+}
+
+func (m *Manager) connectSupportContainersToTenantNetwork(ctx context.Context, networkName string) error {
+	if m == nil || !m.cfg.IsolateTenantNetworks {
+		return nil
+	}
+	for _, label := range m.cfg.SupportContainerLabels {
+		label = strings.TrimSpace(label)
+		if label == "" {
+			continue
+		}
+		if err := m.connectSupportContainersByLabel(ctx, networkName, label); err != nil {
+			return err
+		}
+	}
+	return nil
+}
+
+func (m *Manager) connectSupportContainersByLabel(ctx context.Context, networkName, label string) error {
+	filters := client.Filters{}
+	filters = filters.Add("label", label)
+	result, err := m.cli.ContainerList(ctx, client.ContainerListOptions{
+		Filters: filters,
+	})
+	if err != nil {
+		return fmt.Errorf("list provider support containers for label %q: %w", label, err)
+	}
+	if len(result.Items) == 0 {
+		return fmt.Errorf("provider support container with label %q is required for isolated tenant network %q", label, networkName)
+	}
+
+	for _, item := range result.Items {
+		if item.ID == "" {
+			continue
+		}
+		if err := m.ensureContainerConnectedToNetwork(ctx, networkName, item.ID); err != nil {
+			return fmt.Errorf("connect provider support container %s to tenant network %q: %w", item.ID[:min(len(item.ID), 12)], networkName, err)
+		}
+	}
+	return nil
+}
+
+func (m *Manager) ensureContainerConnectedToNetwork(ctx context.Context, networkName, containerID string) error {
+	inspect, err := m.cli.NetworkInspect(ctx, networkName, client.NetworkInspectOptions{})
+	if err != nil {
+		return fmt.Errorf("inspect tenant network %q: %w", networkName, err)
+	}
+	for id := range inspect.Network.Containers {
+		if id == containerID || strings.HasPrefix(id, containerID) || strings.HasPrefix(containerID, id) {
+			return nil
+		}
+	}
+	_, err = m.cli.NetworkConnect(ctx, networkName, client.NetworkConnectOptions{
+		Container:      containerID,
+		EndpointConfig: &network.EndpointSettings{},
+	})
+	return err
+}
+
 // IsNotFound reports whether Docker treated an identifier as missing.
 func IsNotFound(err error) bool {
 	return errdefs.IsNotFound(err)
@ -328,7 +490,12 @@ func (m *Manager) CreateAndStart(ctx context.Context, tenantID, tenantDataDir st
 		return "", fmt.Errorf("prepare tenant runtime mounts for %s: %w", tenantID, err)
 	}

-	labels := TraefikLabels(tenantID, m.cfg.BaseDomain, m.cfg.ContainerPort)
+	tenantNetworkName, err := m.ensureTenantNetwork(ctx, tenantID)
+	if err != nil {
+		return "", err
+	}
+
+	labels := TraefikLabels(tenantID, m.cfg.BaseDomain, m.cfg.ContainerPort, tenantNetworkName)
 	labels["pulse.managed"] = "true"

 	containerName := "pulse-" + tenantID
@ -337,7 +504,7 @@ func (m *Manager) CreateAndStart(ctx context.Context, tenantID, tenantDataDir st
 		Config: &container.Config{
 			Image:  m.cfg.Image,
 			Labels: labels,
-			Env:    tenantEnv(tenantID, m.cfg.BaseDomain, m.cfg.TrialActivationPublicKey, m.tenantTrustedProxyCIDRs(ctx)),
+			Env:    tenantEnv(tenantID, m.cfg.BaseDomain, m.cfg.TrialActivationPublicKey, m.tenantTrustedProxyCIDRs(ctx, tenantNetworkName)),
 		},
 		HostConfig: &container.HostConfig{
 			RestartPolicy: container.RestartPolicy{Name: "unless-stopped"},
@ -350,7 +517,7 @@ func (m *Manager) CreateAndStart(ctx context.Context, tenantID, tenantDataDir st
 		},
 		NetworkingConfig: &network.NetworkingConfig{
 			EndpointsConfig: map[string]*network.EndpointSettings{
-				m.cfg.Network: {},
+				tenantNetworkName: {},
 			},
 		},
 		Name: containerName,
@ -359,6 +526,11 @@ func (m *Manager) CreateAndStart(ctx context.Context, tenantID, tenantDataDir st
 		return "", fmt.Errorf("create container for %s: %w", tenantID, err)
 	}

+	if err := m.connectSupportContainersToTenantNetwork(ctx, tenantNetworkName); err != nil {
+		_, _ = m.cli.ContainerRemove(ctx, resp.ID, client.ContainerRemoveOptions{Force: true})
+		return resp.ID, err
+	}
+
 	if _, err := m.cli.ContainerStart(ctx, resp.ID, client.ContainerStartOptions{}); err != nil {
 		return resp.ID, fmt.Errorf("start container for %s: %w", tenantID, err)
 	}
@ -460,7 +632,7 @@ func envValue(env []string, key string) string {
 	return ""
 }

-func (m *Manager) tenantTrustedProxyCIDRs(ctx context.Context) []string {
+func (m *Manager) tenantTrustedProxyCIDRs(ctx context.Context, networkNames ...string) []string {
 	values := make([]string, 0, len(m.cfg.TrustedProxyCIDRs)+1)
 	seen := make(map[string]struct{})
 	appendCIDR := func(raw string) {
@ -479,8 +651,14 @@ func (m *Manager) tenantTrustedProxyCIDRs(ctx context.Context) []string {
 		appendCIDR(cidr)
 	}

-	if m != nil && m.cli != nil && strings.TrimSpace(m.cfg.Network) != "" {
-		networkName := strings.TrimSpace(m.cfg.Network)
+	if len(networkNames) == 0 && m != nil {
+		networkNames = []string{m.cfg.Network}
+	}
+	for _, networkName := range networkNames {
+		networkName = strings.TrimSpace(networkName)
+		if m == nil || m.cli == nil || networkName == "" {
+			continue
+		}
 		inspect, err := m.cli.NetworkInspect(ctx, networkName, client.NetworkInspectOptions{})
 		if err != nil {
 			log.Warn().Err(err).Str("network", networkName).Msg("Failed to inspect tenant network for trusted proxy CIDRs")
@ -563,12 +741,101 @@ func (m *Manager) Start(ctx context.Context, containerID string) error {

 // Remove removes a stopped tenant container.
 func (m *Manager) Remove(ctx context.Context, containerID string) error {
+	networkNames := m.tenantNetworkNamesForContainer(ctx, containerID)
 	_, err := m.cli.ContainerRemove(ctx, containerID, client.ContainerRemoveOptions{
 		Force: true,
 	})
+	if err == nil {
+		for _, networkName := range networkNames {
+			if removeErr := m.removeTenantNetwork(ctx, networkName); removeErr != nil {
+				log.Warn().Err(removeErr).Str("network", networkName).Msg("Failed to remove tenant runtime network")
+			}
+		}
+	}
 	return err
 }

+func (m *Manager) tenantNetworkNamesForContainer(ctx context.Context, containerID string) []string {
+	if m == nil || !m.cfg.IsolateTenantNetworks || strings.TrimSpace(containerID) == "" {
+		return nil
+	}
+	inspectResult, err := m.cli.ContainerInspect(ctx, containerID, client.ContainerInspectOptions{})
+	if err != nil {
+		return nil
+	}
+	inspect := inspectResult.Container
+	if inspect.NetworkSettings == nil {
+		return nil
+	}
+	var out []string
+	for networkName := range inspect.NetworkSettings.Networks {
+		if m.isTenantNetwork(ctx, networkName) {
+			out = append(out, networkName)
+		}
+	}
+	sort.Strings(out)
+	return out
+}
+
+func (m *Manager) isTenantNetwork(ctx context.Context, networkName string) bool {
+	networkName = strings.TrimSpace(networkName)
+	if m == nil || networkName == "" {
+		return false
+	}
+	inspect, err := m.cli.NetworkInspect(ctx, networkName, client.NetworkInspectOptions{})
+	if err != nil {
+		return false
+	}
+	return strings.TrimSpace(inspect.Network.Labels[tenantRuntimeNetworkLabel]) == tenantRuntimeNetworkLabelValue
+}
+
+func (m *Manager) removeTenantNetwork(ctx context.Context, networkName string) error {
+	if m == nil || strings.TrimSpace(networkName) == "" {
+		return nil
+	}
+	if err := m.disconnectSupportContainersFromTenantNetwork(ctx, networkName); err != nil {
+		return err
+	}
+	if _, err := m.cli.NetworkRemove(ctx, networkName, client.NetworkRemoveOptions{}); err != nil {
+		return fmt.Errorf("remove tenant network %q: %w", networkName, err)
+	}
+	return nil
+}
+
+func (m *Manager) disconnectSupportContainersFromTenantNetwork(ctx context.Context, networkName string) error {
+	if m == nil || !m.cfg.IsolateTenantNetworks {
+		return nil
+	}
+	for _, label := range m.cfg.SupportContainerLabels {
+		label = strings.TrimSpace(label)
+		if label == "" {
+			continue
+		}
+		filters := client.Filters{}
+		filters = filters.Add("label", label)
+		result, err := m.cli.ContainerList(ctx, client.ContainerListOptions{
+			All:     true,
+			Filters: filters,
+		})
+		if err != nil {
+			return fmt.Errorf("list provider support containers for disconnect label %q: %w", label, err)
+		}
+		for _, item := range result.Items {
+			if item.ID == "" {
+				continue
+			}
+			_, err := m.cli.NetworkDisconnect(ctx, networkName, client.NetworkDisconnectOptions{
+				Container: item.ID,
+				Force:     true,
+			})
+			if err != nil && !errdefs.IsNotFound(err) {
+				return fmt.Errorf("disconnect provider support container %s from tenant network %q: %w", item.ID[:min(len(item.ID), 12)], networkName, err)
+			}
+		}
+	}
+	return nil
+}
+
 // Rename renames a tenant container to the supplied Docker name.
 func (m *Manager) Rename(ctx context.Context, containerIDOrName, newName string) error {
 	if err := m.ensureDaemonReachable(ctx); err != nil {
@ -634,10 +901,17 @@ func (m *Manager) HealthCheck(ctx context.Context, containerID string) (bool, er
 		return false, nil
 	}

-	// Find the container's IP on our network
-	netSettings, ok := inspect.NetworkSettings.Networks[m.cfg.Network]
-	if !ok || !netSettings.IPAddress.IsValid() {
-		return false, fmt.Errorf("container not connected to network %s", m.cfg.Network)
+	networkNames := m.healthCheckNetworkCandidates(inspect)
+	var netSettings *network.EndpointSettings
+	for _, networkName := range networkNames {
+		candidate, ok := inspect.NetworkSettings.Networks[networkName]
+		if ok && candidate != nil && candidate.IPAddress.IsValid() {
+			netSettings = candidate
+			break
+		}
+	}
+	if netSettings == nil {
+		return false, fmt.Errorf("container not connected to tenant health network candidates %s", strings.Join(networkNames, ", "))
 	}

 	healthURL := fmt.Sprintf("http://%s:%d/api/health", netSettings.IPAddress.String(), m.cfg.ContainerPort)
@ -650,3 +924,34 @@ func (m *Manager) HealthCheck(ctx context.Context, containerID string) (bool, er

 	return resp.StatusCode == http.StatusOK, nil
 }
+
+func (m *Manager) healthCheckNetworkCandidates(inspect container.InspectResponse) []string {
+	var names []string
+	seen := make(map[string]struct{})
+	appendName := func(name string) {
+		name = strings.TrimSpace(name)
+		if name == "" {
+			return
+		}
+		if _, ok := seen[name]; ok {
+			return
+		}
+		seen[name] = struct{}{}
+		names = append(names, name)
+	}
+
+	if m != nil && m.cfg.IsolateTenantNetworks && inspect.Config != nil {
+		if tenantID := strings.TrimSpace(inspect.Config.Labels["pulse.tenant.id"]); tenantID != "" {
+			appendName(m.tenantNetworkName(tenantID))
+		}
+	}
+	if m != nil {
+		appendName(m.cfg.Network)
+	}
+	if inspect.NetworkSettings != nil {
+		for name := range inspect.NetworkSettings.Networks {
+			appendName(name)
+		}
+	}
+	return names
+}
--- a/internal/cloudcp/docker/manager_test.go
+++ b/internal/cloudcp/docker/manager_test.go
@ -7,6 +7,9 @@ import (
 	"strings"
 	"syscall"
 	"testing"
+
+	"github.com/moby/moby/api/types/container"
+	mobynetwork "github.com/moby/moby/api/types/network"
 )

 func TestTenantImmutableOwnershipPaths(t *testing.T) {
@ -115,6 +118,66 @@ func TestTraefikLabelsUseCanonicalLowercaseHost(t *testing.T) {
 	}
 }

+func TestTraefikLabelsPinTenantRuntimeToIsolatedNetwork(t *testing.T) {
+	t.Parallel()
+
+	labels := TraefikLabels("t-acme", "msp.example.com", 7655, "pulse-provider-msp-tenant-t-acme")
+	if got := labels["traefik.docker.network"]; got != "pulse-provider-msp-tenant-t-acme" {
+		t.Fatalf("traefik.docker.network = %q, want isolated tenant network", got)
+	}
+}
+
+func TestTenantNetworkNameIsDerivedPerTenant(t *testing.T) {
+	t.Parallel()
+
+	mgr := &Manager{cfg: ManagerConfig{
+		Network:               "pulse-provider-msp",
+		IsolateTenantNetworks: true,
+		TenantNetworkPrefix:   "pulse-provider-msp-tenant",
+	}}
+
+	gotA := mgr.tenantNetworkName("T-Acme")
+	gotB := mgr.tenantNetworkName("T-Beta")
+	if gotA != "pulse-provider-msp-tenant-t-acme" {
+		t.Fatalf("tenantNetworkName(T-Acme) = %q", gotA)
+	}
+	if gotB != "pulse-provider-msp-tenant-t-beta" {
+		t.Fatalf("tenantNetworkName(T-Beta) = %q", gotB)
+	}
+	if gotA == gotB {
+		t.Fatalf("tenant networks must be distinct, got %q", gotA)
+	}
+}
+
+func TestHealthCheckPrefersProviderMSPIsolatedTenantNetwork(t *testing.T) {
+	t.Parallel()
+
+	mgr := &Manager{cfg: ManagerConfig{
+		Network:               "pulse-provider-msp",
+		IsolateTenantNetworks: true,
+		TenantNetworkPrefix:   "pulse-provider-msp-tenant",
+	}}
+	inspect := container.InspectResponse{
+		Config: &container.Config{
+			Labels: map[string]string{"pulse.tenant.id": "t-acme"},
+		},
+		NetworkSettings: &container.NetworkSettings{
+			Networks: map[string]*mobynetwork.EndpointSettings{
+				"pulse-provider-msp":               {},
+				"pulse-provider-msp-tenant-t-acme": {},
+			},
+		},
+	}
+
+	got := mgr.healthCheckNetworkCandidates(inspect)
+	if len(got) < 2 {
+		t.Fatalf("healthCheckNetworkCandidates = %v, want tenant and ingress networks", got)
+	}
+	if got[0] != "pulse-provider-msp-tenant-t-acme" {
+		t.Fatalf("first health network = %q, want isolated tenant network; all=%v", got[0], got)
+	}
+}
+
 func TestTenantEnvLowercasesPublicURLHost(t *testing.T) {
 	t.Parallel()

--- a/internal/cloudcp/registry/registry_test.go
+++ b/internal/cloudcp/registry/registry_test.go
@ -3,9 +3,11 @@ package registry
 import (
 	"database/sql"
 	"errors"
+	"fmt"
 	"os"
 	"path/filepath"
 	"strings"
+	"sync"
 	"testing"
 	"time"

@ -957,6 +959,63 @@ func TestCreateWithAccountWorkspaceLimitEnforcesActiveCount(t *testing.T) {
 	}
 }

+func TestCreateWithAccountWorkspaceLimitCannotRacePastLimit(t *testing.T) {
+	reg := newTestRegistry(t)
+
+	accountID, err := GenerateAccountID()
+	if err != nil {
+		t.Fatal(err)
+	}
+	if err := reg.CreateAccount(&Account{ID: accountID, Kind: AccountKindMSP, DisplayName: "Race Account"}); err != nil {
+		t.Fatal(err)
+	}
+
+	const limit = 2
+	const attempts = 16
+	var wg sync.WaitGroup
+	errs := make(chan error, attempts)
+	for i := 0; i < attempts; i++ {
+		wg.Add(1)
+		go func(i int) {
+			defer wg.Done()
+			errs <- reg.CreateWithAccountWorkspaceLimit(&Tenant{
+				ID:        fmt.Sprintf("t-RACE%04d", i),
+				AccountID: accountID,
+				State:     TenantStateProvisioning,
+			}, limit)
+		}(i)
+	}
+	wg.Wait()
+	close(errs)
+
+	successes := 0
+	limitErrors := 0
+	for err := range errs {
+		if err == nil {
+			successes++
+			continue
+		}
+		if _, ok := WorkspaceLimitExceeded(err); ok {
+			limitErrors++
+			continue
+		}
+		t.Fatalf("unexpected create error: %v", err)
+	}
+	if successes != limit {
+		t.Fatalf("successful creates = %d, want exactly limit %d", successes, limit)
+	}
+	if limitErrors != attempts-limit {
+		t.Fatalf("limit errors = %d, want %d", limitErrors, attempts-limit)
+	}
+	tenants, err := reg.ListByAccountID(accountID)
+	if err != nil {
+		t.Fatalf("ListByAccountID: %v", err)
+	}
+	if len(tenants) != limit {
+		t.Fatalf("tenant count = %d, want %d", len(tenants), limit)
+	}
+}
+
 func TestCountByState(t *testing.T) {
 	reg := newTestRegistry(t)

--- a/internal/cloudcp/server.go
+++ b/internal/cloudcp/server.go
@ -55,6 +55,7 @@ func Run(ctx context.Context, version string) error {
 	dockerMgr, err = cpDocker.NewManager(cpDocker.ManagerConfig{
 		Image:                    cfg.PulseImage,
 		Network:                  cfg.DockerNetwork,
+		IsolateTenantNetworks:    cfg.IsProviderHostedMSP(),
 		BaseDomain:               baseDomainFromURL(cfg.BaseURL),
 		TrialActivationPublicKey: cfg.TrialActivationPublicKey,
 		TrustedProxyCIDRs:        cfg.TrustedProxyCIDRs,
--- a/internal/cloudcp/tenant_runtime_rollout.go
+++ b/internal/cloudcp/tenant_runtime_rollout.go
@ -218,6 +218,7 @@ func newTenantRuntimeRolloutServiceFromConfig(
 	dockerMgr, err := cpDocker.NewManager(cpDocker.ManagerConfig{
 		Image:                    image,
 		Network:                  cfg.DockerNetwork,
+		IsolateTenantNetworks:    cfg.IsProviderHostedMSP(),
 		BaseDomain:               baseDomainFromURL(cfg.BaseURL),
 		TrialActivationPublicKey: cfg.TrialActivationPublicKey,
 		TrustedProxyCIDRs:        cfg.TrustedProxyCIDRs,
--- a/scripts/installtests/provider_msp_deploy_test.go
+++ b/scripts/installtests/provider_msp_deploy_test.go
@ -28,6 +28,8 @@ func TestProviderMSPDeployComposeIsProviderModeAndStripeFree(t *testing.T) {
 		"CP_STORAGE_DATA_PATH=${PULSE_PROVIDER_MSP_DATA_DIR:-/data}",
 		"${PULSE_PROVIDER_MSP_DATA_DIR:-/data}:${PULSE_PROVIDER_MSP_DATA_DIR:-/data}",
 		"${PULSE_PROVIDER_MSP_DOCKER_SOCKET:-/var/run/docker.sock}:/var/run/docker.sock",
+		"pulse.provider-msp.role=traefik",
+		"pulse.provider-msp.role=control-plane",
 		"provider_msp_license:",
 		"name: ${PULSE_PROVIDER_MSP_DOCKER_NETWORK:-pulse-provider-msp}",
 	)
@ -195,6 +197,7 @@ func TestProviderMSPInstallProofRunnerMatchesComposeContract(t *testing.T) {
 		"docker compose config --quiet",
 		"docker version >/dev/null",
 		"docker compose pull traefik control-plane",
+		"docker compose up -d traefik",
 		"provider-msp install-proof",
 		"--account-name",
 		"--owner-email",