docs: codify the release confidence process with executable tooling (#1052)

Captures the process designed and executed for v1.11.0 so every future release reproduces it: - .github/RELEASE_PROCESS.md v2: changelog audit, risk-based test matrix (buckets A/B/C), the Docker image gate, fix-loop re-test policy, CI-based publishing path, communication structure with a mandatory credits section, retro, and the gotchas that cost iterations this cycle - ADR-005: why releases now pass a risk-based confidence process gated on the real image, with the v1.11.0 evidence (bugs the unit suite could not catch: SEARCH-index ORDER BY 500, credential clear no-op) - scripts/release-test/: the harness built during v1.11.0 — fresh-install + upgrade gate (release-image-test.sh) and the browsable RC stack with optional dev-data copy (rc-stack.sh), plus compose/nginx encoding the API_URL, host.docker.internal and SurrealDB import learnings - make release-test / release-stack / release-stack-down targets
2026-07-24 07:05:47 +00:00 · 2026-07-11 16:56:14 -03:00 · 2026-07-11 16:56:14 -03:00 · faa2b16e77
commit faa2b16e77
parent d9ad3917df
8 changed files with 496 additions and 40 deletions
--- a/.github/RELEASE_PROCESS.md
+++ b/.github/RELEASE_PROCESS.md
@ -4,15 +4,18 @@ Open Notebook uses a flow-driven release process. Work moves from `ready`
 issues into pull requests, pull requests merge to `main`, and maintainers cut a
 version when the branch has enough validated change to ship.

+This document covers both the **mechanics** (how to cut, build and publish) and
+the **confidence process** (how we know a release is good before users get it).
+It was redesigned during the v1.11.0 release ([ADR-005](../docs/7-DEVELOPMENT/decisions/ADR-005-release-confidence-process.md)).
+
 ## Release Model

 - Patch releases ship backwards-compatible fixes.
 - Minor releases ship backwards-compatible features and improvements.
 - Major releases are planned with a milestone when they include breaking
  changes or migrations that need user coordination.
- Release candidates and community soak labels are no longer part of the
-  process. Use the `in-dev-build` label for changes available in development
-  images and `released` for shipped work.
+- Use the `in-dev-build` label for changes available in development images and
+  `released` for shipped work.

 ## Normal Flow

@ -21,59 +24,132 @@ version when the branch has enough validated change to ship.
 3. Merge the pull request after review and required checks pass.
 4. Let the development build publish the `v1-dev` image from `main`.
 5. Cut a stable release when `main` has a coherent set of changes ready for
-   users.
+   users — following the confidence process below.
+
+## The Confidence Process
+
+Releases keep getting bigger; ad-hoc verification does not scale. Before
+cutting, run this sequence:
+
+### 0. Changelog audit
+
+Diff `git log <last-tag>..main` against the `[Unreleased]` section of the
+CHANGELOG. Every merged PR must be represented (entries reference the issue
+number when one exists, the PR number otherwise). The changelog is the input
+for both the test plan and the release notes — close the gaps first, via PR.
+
+### 1. Risk-based test matrix
+
+Build a matrix from the actual release diff: each change → what it can break
+and for whom → which bucket tests it. Pay special attention to
+**"does the protection break legitimate use?"** for security changes (e.g. an
+SSRF guard vs. self-hosted Ollama on localhost) and to anything a reverse
+proxy, an upgrade, or a big upload would exercise.
+
+Buckets:
+
+- **A — automated, high confidence, run now**: full backend suite, frontend
+  lint/tests/production build, the smoke-e2e agent (full API happy path + UI
+  verification), targeted regression probes for the release's specific risks,
+  dependency audit.
+- **B — automatable with investment**: decide per item whether to build the
+  muscle now (it compounds: the image gate below started as a bucket-B item)
+  or verify manually this once.
+- **C — needs the release owner**: real provider credentials, real TTS podcast
+  generation, visual/UX judgment, and the final check of the pushed image.
+
+### 2. The image gate — test the artifact, not the repo
+
+A green suite on `main` is not a working image. Run:
+
+```bash
+make docker-build-local          # builds <version> + local tags
+make release-test TAG=<new> OLD_TAG=<previous>
+```
+
+This runs two scenarios against real containers (`scripts/release-test/`):
+
+- **Fresh install**: empty DB → migrations on boot → in-image worker processes
+  a source → API/frontend/nginx-proxied checks.
+- **Upgrade**: boot the *published* previous image, seed data, swap to the new
+  image on the same volume → migrations apply, data survives.
+
+Caveat: `docker-build-local` tags with the current `pyproject.toml` version —
+`docker pull` the genuine previous tag before the upgrade test so you are not
+comparing the new build against itself.
+
+### 3. Fix loop with a re-test policy
+
+Findings become focused PRs through the normal review flow. After each merge:
+the cheap suite always re-runs; smoke/image gates re-run only if the fix
+touches what they cover; manual verification is not repeated unless the fix
+touches what was manually verified. Pre-existing bugs found along the way that
+are not release regressions become backlog issues instead of scope creep.

 ## Cutting A Stable Release

-1. Confirm `main` is green and review the changes since the previous release.
-2. Update `pyproject.toml` with the target semantic version.
-3. Move the relevant `CHANGELOG.md` entries from `Unreleased` into a dated
-   version section.
-4. Create a GitHub release for the version tag, for example `v1.11.0`.
-5. Run the `Build and Release` workflow. Push `v1-latest` tags for normal
-   stable releases.
-6. Verify the published images are available in GHCR and Docker Hub when
-   Docker Hub credentials are configured.
-7. Mark shipped issues with `released` and close any release-tracking tasks.
+1. Confirm `main` is green and the confidence process above has run.
+2. Open the **cut PR**: bump `pyproject.toml`, date the `[Unreleased]` section
+   as `[<version>] - <date>`.
+3. After merge: `make tag`.
+4. Build and push version images **via CI** (it holds the registry
+   credentials): trigger the *Build and Release* workflow with
+   `push_latest=false`. Local `make docker-push` also works but requires
+   `docker login` on both registries.
+5. **Verify the pushed image** (bucket C, final gate): run it locally with
+   `make release-stack TAG=<version> [DUMP=<dev-data-dump>]` — a browsable,
+   isolated stack, optionally with a copy of real data — and walk the core
+   flows in the browser.
+6. Publish the GitHub release. A non-prerelease publication triggers the
+   workflow again and pushes the `v1-latest` tags automatically.
+7. Verify the `v1-latest` manifests on Docker Hub and GHCR (both arches, both
+   variants), and mark shipped issues with `released`.

-## Docker Image Publishing
+## Communication

-The publishing workflow, from local test to promoted release:
+Release notes follow this structure (see v1.11.0 as the reference):

-```bash
-make docker-build-local   # 1. Build for the current platform, test locally
-make docker-push          # 2. Push version tags (does NOT update latest)
-                          # 3. Test the pushed version in staging/production
-make docker-push-latest   # 4. Promote: push version + update v1-latest
-```
+1. One-line verdict + upgrade recommendation.
+2. Sections: Security, Features, Performance, Notable fixes.
+3. **Behavior changes for self-hosters** — anything that can require a config
+   tweak on upgrade gets an explicit callout.
+4. **Thanks** — credit every contributor by handle with what they shipped
+   (collect via `git log <last-tag>..<tag>` + `gh pr view` for handles), plus
+   the issue reporters collectively. Never skip this section.
+
+Announce on Discord after `v1-latest` is live.
+
+## Retro
+
+Close every release by asking: what should improve in this process? Apply the
+accepted improvements immediately — update this document, the scripts under
+`scripts/release-test/`, and the decision log while the context is fresh.
+
+## Docker Image Publishing (reference)

 | Command | What it does | Updates latest? |
 |---------|--------------|-----------------|
 | `make docker-build-local` | Build for current platform only (tags `<version>` + `local`) | No registry push |
-| `make docker-push` | Push version tags to registries | ❌ No |
-| `make docker-push-latest` | Push version + update `v1-latest` | ✅ Yes |
-| `make docker-release` | Full release (same as docker-push-latest) | ✅ Yes |
+| CI *Build and Release* (`push_latest=false`) | Push version tags via CI credentials | ❌ No |
+| GitHub release published (non-prerelease) | CI pushes version + `v1-latest` | ✅ Yes |
+| `make docker-push` / `docker-push-latest` | Local equivalents (need `docker login`) | ❌ / ✅ |
 | `make tag` | Create and push a git tag matching `pyproject.toml` | — |

-Publishing details:
-
 - **Platforms:** `linux/amd64`, `linux/arm64`
 - **Registries:** Docker Hub + GitHub Container Registry
 - **Image variants:** regular + single-container (`-single`)
 - **Version source:** `pyproject.toml`
 - Build issues: `docker builder prune`, then `make docker-buildx-reset`

-## Manual Verification
+## Known Gotchas

-Before publishing a stable release, manually verify the areas touched by the
-release. At minimum, cover:
-
- installation or upgrade path changed by the release;
- source ingestion and processing when content or worker behavior changed;
- chat, search, notes, and notebooks when user workflows changed;
- provider setup when model, credential, or API behavior changed;
- Docker image startup when packaging, environment, or dependency changes
-  landed.
-
-Automated checks should catch regressions where possible, but the release owner
-chooses the manual matrix from the actual changes in the release.
+- **RC stack on non-default ports needs `API_URL`** or the browser talks to
+  `host:5055` — on a dev machine that is the development API (data crossover).
+  `rc-stack.sh` sets it; remember this for any custom setup.
+- **Containerized app + host services**: credentials pointing at local
+  services (Ollama, LM Studio) need `http://host.docker.internal:<port>`.
+- **SurrealDB import**: `OVERWRITE` goes after the type keyword
+  (`DEFINE FIELD OVERWRITE …`), and the exporter can leak a log line into the
+  dump — `rc-stack.sh` handles both.
+- **Multiple local SurrealDB instances**: check which one the dev `.env`
+  actually points at (`SURREAL_URL`) before exporting data.
--- a/CHANGELOG.md
+++ b/CHANGELOG.md
@ -5,6 +5,11 @@ All notable changes to this project will be documented in this file.
 The format is based on [Keep a Changelog](https://keepachangelog.com/en/1.1.0/),
 and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0.html).

+## [Unreleased]
+
+### Added
+- Release confidence process, documented and executable: `.github/RELEASE_PROCESS.md` now covers the risk-based test matrix (buckets A/B/C), the Docker image gate, the fix-loop re-test policy and the communication/credits/retro structure, backed by a new decision record (ADR-005) and versioned tooling under `scripts/release-test/` — `make release-test TAG= OLD_TAG=` runs fresh-install + upgrade scenarios against real images, and `make release-stack TAG= [DUMP=]` boots a browsable, isolated release-candidate stack (optionally with a copy of dev data) for manual verification
+
 ## [1.11.0] - 2026-07-11

 ### Added
--- a/20
+++ b/20
@ -1,6 +1,7 @@
 .PHONY: run frontend check ruff database lint api start-all stop-all status clean-cache worker worker-start worker-stop worker-restart
 .PHONY: docker-buildx-prepare docker-buildx-clean docker-buildx-reset
 .PHONY: docker-push docker-push-latest docker-release docker-build-local tag export-docs
+.PHONY: release-test release-stack release-stack-down

 # Get version from pyproject.toml
 VERSION := $(shell grep -m1 version pyproject.toml | cut -d'"' -f2)
@ -43,6 +44,25 @@ docker-buildx-clean:
 docker-buildx-reset: docker-buildx-clean docker-buildx-prepare
 	@echo "✅ Buildx reset complete!"

+# === Release Testing (see .github/RELEASE_PROCESS.md) ===
+
+# Automated image gate: fresh install + upgrade against real images.
+# Usage: make release-test TAG=1.12.0 OLD_TAG=1.11.0
+release-test:
+	@test -n "$(TAG)" || (echo "usage: make release-test TAG=<new> [OLD_TAG=<previous>]"; exit 1)
+	bash scripts/release-test/release-image-test.sh all \
+		"$(DOCKERHUB_IMAGE):$(TAG)" \
+		$(if $(OLD_TAG),"$(DOCKERHUB_IMAGE):$(OLD_TAG)")
+
+# Browsable RC stack for manual verification (optionally with a data dump).
+# Usage: make release-stack TAG=1.12.0 [DUMP=/tmp/dev-dump.surql]
+release-stack:
+	@test -n "$(TAG)" || (echo "usage: make release-stack TAG=<tag> [DUMP=<dump.surql>]"; exit 1)
+	bash scripts/release-test/rc-stack.sh up "$(TAG)" $(DUMP)
+
+release-stack-down:
+	bash scripts/release-test/rc-stack.sh down "$(or $(TAG),unused)"
+
 # === Docker Build Targets ===

 # Build production image for local platform only (no push)
--- a/docs/7-DEVELOPMENT/decisions/ADR-005-release-confidence-process.md
+++ b/docs/7-DEVELOPMENT/decisions/ADR-005-release-confidence-process.md
@ -0,0 +1,69 @@
+# ADR-005: Releases pass a risk-based confidence process, gated on the real image
+
+- **Status**: Accepted
+- **Date**: 2026-07 (established during the v1.11.0 release)
+- **Related**: [RELEASE_PROCESS.md](../../../.github/RELEASE_PROCESS.md), `scripts/release-test/`, [ADR-004](ADR-004-background-workers.md)
+
+## Context
+
+Releases grew from a handful of fixes to 50+ commits spanning security
+hardening, features, migrations and dependency changes. Verification was
+ad-hoc: a green test suite on `main` plus whatever manual checks the release
+owner remembered. v1.11.0 proved the gap empirically — the unit suite was
+fully green while `sort_by=title` returned a 500 (a SEARCH-index interaction
+only a real SurrealDB exhibits) and clearing credential fields silently
+no-oped (two mirror-image bugs, frontend and API, that only an end-to-end
+path reveals). Neither class of bug is catchable by mocked tests, and neither
+was: both were found by the process this record establishes.
+
+## Decision
+
+**Every stable release passes a risk-based confidence process before cutting,
+and the final gate runs against the built Docker image — the artifact users
+receive — not the repository.**
+
+The process (mechanics in [RELEASE_PROCESS.md](../../../.github/RELEASE_PROCESS.md)):
+
+1. **Changelog audit first** — the release diff, fully represented in the
+   CHANGELOG, is the input for both the test plan and the communication.
+2. **Risk matrix over test list** — each change is classified by what it can
+   break and for whom, then assigned to a bucket: **A** (automated now),
+   **B** (automatable with investment — build the muscle when it compounds),
+   **C** (release-owner judgment: real credentials, real TTS, UX, the pushed
+   image). Security changes are probed for the inverse risk: does the
+   protection break legitimate use?
+3. **The image gate** — fresh-install and upgrade-with-data scenarios run
+   against real containers (`make release-test`), because packaging bugs
+   (supervisord flags, uv sync modes, migration ordering) never appear in the
+   suite.
+4. **Fix loop with a re-test policy** — findings become focused PRs; what
+   re-runs after each merge is defined up front. Pre-existing bugs that are
+   not release regressions become backlog issues, not scope creep.
+5. **Human gates stay human** — the pushed-image verification and the release
+   publication require the release owner explicitly; automation prepares,
+   people pull the trigger.
+6. **Retro closes the loop** — accepted improvements are applied to the
+   process docs and scripts in the same session.
+
+## Alternatives considered
+
+- **Keep ad-hoc verification** — free, but v1.11.0 showed it misses exactly
+  the bug classes that hurt users most (integration and packaging).
+- **Full CI-based E2E on every PR** — highest coverage, but a real
+  SurrealDB + worker + image build pipeline on every PR is slow and expensive;
+  the release boundary is where artifact-level confidence pays off.
+- **Community soak (RC tags)** — previously abandoned: slow feedback and low
+  participation; a deliberate confidence process front-loads what soaking
+  found late.
+
+## Consequences
+
+- Cutting a release costs hours, not minutes — deliberately: the cost scales
+  with release size, which is the point of the risk matrix.
+- Release muscle is versioned in-repo (`scripts/release-test/`, make targets)
+  and compounds: bucket-B investments from one release become bucket-A
+  automation for the next.
+- The upgrade scenario requires published previous images to remain available
+  on the registries.
+- The process assumes a release owner in the loop for buckets B/C — it is a
+  confidence process, not full automation.
--- a/scripts/release-test/docker-compose.release-test.yml
+++ b/scripts/release-test/docker-compose.release-test.yml
@ -0,0 +1,43 @@
+services:
+  surrealdb:
+    image: surrealdb/surrealdb:v2
+    command: start --log info --user root --pass root rocksdb:/mydata/test.db
+    user: root
+    volumes:
+      - ${DATA_DIR:?set DATA_DIR}/surreal:/mydata
+
+  app:
+    image: ${APP_IMAGE:?set APP_IMAGE}
+    ports:
+      - "127.0.0.1:${API_PORT:?set API_PORT}:5055"
+      - "127.0.0.1:${FE_PORT:?set FE_PORT}:8502"
+    environment:
+      # Without an explicit API_URL the frontend's /config points the BROWSER
+      # at host:5055 — on a dev machine that is the development API, not this
+      # stack (silent data crossover between environments!)
+      - API_URL=${RC_API_URL:-}
+      - OPEN_NOTEBOOK_ENCRYPTION_KEY=${RC_ENCRYPTION_KEY:-release-test-key}
+      - SURREAL_URL=ws://surrealdb:8000/rpc
+      - SURREAL_USER=root
+      - SURREAL_PASSWORD=root
+      - SURREAL_NAMESPACE=${RC_SURREAL_NS:-open_notebook}
+      - SURREAL_DATABASE=${RC_SURREAL_DB:-open_notebook}
+    volumes:
+      - ${DATA_DIR}/notebook:/app/data
+    # Reach host services (e.g. Ollama on localhost:11434) via
+    # host.docker.internal on Linux too; Docker Desktop resolves it natively.
+    # Credentials pointing at local services must use
+    # http://host.docker.internal:<port> when the app runs in a container.
+    extra_hosts:
+      - "host.docker.internal:host-gateway"
+    depends_on:
+      - surrealdb
+
+  proxy:
+    image: nginx:alpine
+    ports:
+      - "127.0.0.1:${PROXY_PORT:?set PROXY_PORT}:80"
+    volumes:
+      - ./nginx.conf:/etc/nginx/conf.d/default.conf:ro
+    depends_on:
+      - app
--- a/scripts/release-test/nginx.conf
+++ b/scripts/release-test/nginx.conf
@ -0,0 +1,18 @@
+# Reverse proxy in front of the frontend with default buffering ON — the SSE
+# streaming fix must work anyway because the API sends X-Accel-Buffering: no,
+# which nginx honors. Host is passed through verbatim to exercise the
+# frontend's Host validation with a real proxy in the path.
+server {
+    listen 80;
+    server_name _;
+
+    location / {
+        proxy_pass http://app:8502;
+        proxy_http_version 1.1;
+        proxy_set_header Host $http_host;
+        proxy_set_header X-Forwarded-Proto $scheme;
+        proxy_set_header Upgrade $http_upgrade;
+        proxy_set_header Connection "upgrade";
+        proxy_read_timeout 300s;
+    }
+}
--- a/scripts/release-test/rc-stack.sh
+++ b/scripts/release-test/rc-stack.sh
@ -0,0 +1,75 @@
+#!/bin/bash
+# Browsable release-candidate stack on the local machine: runs a pushed (or
+# local) image, optionally with a COPY of your dev data, for manual release
+# verification without touching the dev environment.
+#
+# Usage:
+#   rc-stack.sh up <tag> [dump.surql]   # start (imports the dump if given)
+#   rc-stack.sh down <tag>              # stop and remove everything
+#
+# To produce a data copy from a running dev SurrealDB (originals untouched):
+#   docker exec <surreal-container> /surreal export \
+#     --conn http://localhost:8000 --user root --pass root \
+#     --ns open_notebook --db <your-db> /dev/stdout > /tmp/dev-dump.surql
+#
+# Ports: UI http://localhost:18502 · via nginx http://localhost:18080 · API 15055
+set -euo pipefail
+DIR="$(cd "$(dirname "$0")" && pwd)"
+REPO="$(cd "$DIR/../.." && pwd)"
+TAG="${2:?usage: rc-stack.sh <up|down> <tag> [dump.surql]}"
+DUMP="${3:-}"
+RC_DATA=/tmp/onrel-rc-data
+
+# Reuse the dev encryption key so copied credentials decrypt; the DB name must
+# match the one the dump came from (defaults to the dev .env's database).
+KEY=$(grep '^OPEN_NOTEBOOK_ENCRYPTION_KEY' "$REPO/.env" 2>/dev/null | cut -d= -f2- | tr -d '"' || true)
+DB=$(grep '^SURREAL_DATABASE' "$REPO/.env" 2>/dev/null | cut -d= -f2- | tr -d '"' || true)
+
+compose() {
+  APP_IMAGE="lfnovo/open_notebook:$TAG" DATA_DIR="$RC_DATA" \
+  API_PORT=15055 FE_PORT=18502 PROXY_PORT=18080 \
+  RC_API_URL="http://localhost:15055" \
+  RC_ENCRYPTION_KEY="${KEY:-release-test-key}" RC_SURREAL_DB="${DB:-open_notebook}" \
+  docker compose -p onrelrc -f "$DIR/docker-compose.release-test.yml" "$@"
+}
+
+case "$1" in
+  down)
+    compose down -v
+    rm -rf "$RC_DATA"
+    echo "RC stack removed."
+    ;;
+  up)
+    compose down -v >/dev/null 2>&1 || true
+    rm -rf "$RC_DATA"; mkdir -p "$RC_DATA/surreal" "$RC_DATA/notebook"
+    if [ -n "$DUMP" ]; then
+      # SurrealDB import quirks, learned in production:
+      # - OVERWRITE goes AFTER the type keyword (DEFINE FIELD OVERWRITE ...),
+      #   needed because RELATION tables auto-define in/out fields
+      # - the exporter can leak its own log line into the dump (starts with ESC)
+      sed -E $'/^\x1b/d; s/^DEFINE (TABLE|FIELD|INDEX|ANALYZER|FUNCTION|EVENT|PARAM|ACCESS) /DEFINE \\1 OVERWRITE /' "$DUMP" > "$RC_DATA/surreal/dump.surql"
+    fi
+    compose up -d surrealdb
+    sleep 4
+    if [ -n "$DUMP" ]; then
+      docker exec onrelrc-surrealdb-1 /surreal import --conn http://localhost:8000 \
+        --user root --pass root --ns open_notebook --db "${DB:-open_notebook}" \
+        /mydata/dump.surql
+    fi
+    compose up -d
+    echo "Waiting for API..."
+    for i in $(seq 1 30); do
+      curl -sf -m 5 -o /dev/null http://localhost:15055/docs && break
+      sleep 5
+    done
+    NB=$(curl -s http://localhost:15055/api/notebooks | python3 -c "import json,sys; print(len(json.load(sys.stdin)))" 2>/dev/null || echo "?")
+    echo "RC stack up — image lfnovo/open_notebook:$TAG"
+    echo "  UI:        http://localhost:18502"
+    echo "  via nginx: http://localhost:18080"
+    echo "  API:       http://localhost:15055"
+    echo "  notebooks: $NB"
+    echo
+    echo "NOTE: credentials pointing at host services (e.g. Ollama) need"
+    echo "      base_url http://host.docker.internal:<port> inside the container."
+    ;;
+esac
--- a/scripts/release-test/release-image-test.sh
+++ b/scripts/release-test/release-image-test.sh
@ -0,0 +1,150 @@
+#!/bin/bash
+# Release image gate: fresh-install and upgrade tests against built Docker images.
+#
+# Usage: release-image-test.sh <fresh|upgrade|all> <new-image> [old-image]
+#   e.g. release-image-test.sh all lfnovo/open_notebook:1.12.0 lfnovo/open_notebook:1.11.0
+#
+# IMPORTANT: `make docker-build-local` tags the build with the CURRENT pyproject
+# version. If that version matches a published release, `docker pull` the
+# genuine old tag first or the upgrade test will compare the new build against
+# itself.
+set -u
+DIR="$(cd "$(dirname "$0")" && pwd)"
+COMPOSE="$DIR/docker-compose.release-test.yml"
+NEW_IMAGE="${2:?usage: release-image-test.sh <fresh|upgrade|all> <new-image> [old-image]}"
+OLD_IMAGE="${3:-}"
+PASS=0; FAIL=0
+
+ok()   { echo "  ✅ $1"; PASS=$((PASS+1)); }
+bad()  { echo "  ❌ $1"; FAIL=$((FAIL+1)); }
+check() { if [ "$2" = "$3" ]; then ok "$1"; else bad "$1 (expected=$2, got=$3)"; fi; }
+
+# Each phase gets its own ports so a leaked container can never answer for the
+# wrong stack (learned the hard way: a leftover fresh-install app once served
+# the whole upgrade test).
+set_ports() { API_PORT=$1; FE_PORT=$2; PROXY_PORT=$3; API="http://localhost:$1"; FE="http://localhost:$2"; PROXY="http://localhost:$3"; }
+
+compose_env() { env APP_IMAGE="$1" DATA_DIR="$2" API_PORT="$API_PORT" FE_PORT="$FE_PORT" PROXY_PORT="$PROXY_PORT" docker compose -p "$3" -f "$COMPOSE" "${@:4}"; }
+
+compose_up() { # <image> <datadir> <project>
+  local OUT
+  OUT=$(compose_env "$1" "$2" "$3" up -d --quiet-pull 2>&1)
+  if echo "$OUT" | grep -qi "error"; then
+    bad "compose up ($3): $(echo "$OUT" | grep -i error | head -1)"
+    return 1
+  fi
+  local RUNNING
+  RUNNING=$(docker inspect "$3-app-1" --format '{{.State.Running}}' 2>/dev/null)
+  check "container $3-app-1 running" "true" "$RUNNING"
+}
+
+compose_down() { # <project> <datadir-to-remove (optional)>
+  compose_env unused /tmp/unused "$1" down -v >/dev/null 2>&1
+  if docker ps --format '{{.Names}}' | grep -q "^$1-"; then
+    bad "teardown of $1 left containers running"
+  fi
+  [ -n "${2:-}" ] && rm -rf "$2"
+}
+
+wait_api() { # up to 150s (startup has ~50s of DB-wait backoff + migrations)
+  for i in $(seq 1 30); do
+    if curl -sf -m 5 -o /dev/null "$API/docs"; then return 0; fi
+    sleep 5
+  done
+  return 1
+}
+
+seed_and_verify() {
+  NB=$(curl -s -X POST "$API/api/notebooks" -H "Content-Type: application/json" \
+    -d '{"name":"release-probe","description":"release test seed"}' | python3 -c "import json,sys; print(json.load(sys.stdin)['id'])" 2>/dev/null)
+  [ -z "$NB" ] && { bad "create notebook"; return 1; }
+  ok "create notebook ($NB)"
+  SRC=$(curl -s -X POST "$API/api/sources" -F "type=text" -F "notebooks=[\"$NB\"]" \
+    -F "content=Release test content. The Turing test evaluates machine intelligence." \
+    -F "title=release-probe-source" -F "async_processing=true" -F "embed=false" \
+    | python3 -c "import json,sys; print(json.load(sys.stdin)['id'])" 2>/dev/null)
+  [ -z "$SRC" ] && { bad "create source"; return 1; }
+  ok "create source ($SRC)"
+  ST=""
+  for i in $(seq 1 24); do
+    ST=$(curl -s "$API/api/sources/$SRC/status" | python3 -c "import json,sys; print(json.load(sys.stdin).get('status',''))" 2>/dev/null)
+    [ "$ST" = "completed" ] && break
+    sleep 5
+  done
+  check "in-image worker processed source" "completed" "$ST"
+  FT=$(curl -s "$API/api/sources/$SRC" | python3 -c "import json,sys; d=json.load(sys.stdin); print('yes' if d.get('full_text') else 'no')" 2>/dev/null)
+  check "full_text present" "yes" "$FT"
+}
+
+fresh_test() {
+  echo "═══ FRESH INSTALL — $NEW_IMAGE"
+  set_ports 15055 18502 18080
+  local DD; DD=$(mktemp -d /tmp/onrel-fresh-XXXX)
+  compose_up "$NEW_IMAGE" "$DD" onrelfresh || { compose_down onrelfresh "$DD"; return 1; }
+  if wait_api; then ok "API up (migrations ran on boot)"; else
+    bad "API did not come up in 150s"; docker logs onrelfresh-app-1 2>&1 | tail -20
+    compose_down onrelfresh "$DD"; return 1
+  fi
+  N=$(curl -s "$API/api/notebooks" | python3 -c "import json,sys; print(len(json.load(sys.stdin)))" 2>/dev/null)
+  check "GET /api/notebooks on a virgin database" "0" "$N"
+  seed_and_verify
+  for f in type title created updated insights_count embedded; do
+    S=$(curl -s -o /dev/null -w '%{http_code}' "$API/api/sources?sort_by=$f&limit=2")
+    check "sort_by=$f" "200" "$S"
+  done
+  S=$(curl -s -o /dev/null -w '%{http_code}' -m 15 "$FE")
+  if [ "$S" = "200" ] || [ "$S" = "307" ]; then ok "frontend responds ($S)"; else bad "frontend ($S)"; fi
+  C=$(curl -s -m 10 "$PROXY/config")
+  echo "$C" | grep -q apiUrl && ok "/config via nginx: $C" || bad "/config via nginx: $C"
+  # Malformed Host: nginx may reject with 400 (correct) or the app falls back
+  # (200). What must never happen is a 5xx.
+  C=$(curl -s -m 10 -H 'Host: bad<host>!' "$PROXY/config" -o /dev/null -w '%{http_code}')
+  if [ "$C" = "200" ] || [ "$C" = "400" ]; then ok "/config via nginx with malformed Host -> $C (no 5xx)"; else bad "/config malformed Host -> $C"; fi
+  compose_down onrelfresh "$DD"
+  echo
+}
+
+upgrade_test() {
+  if [ -z "$OLD_IMAGE" ]; then echo "═══ UPGRADE skipped: no old image given"; return 0; fi
+  echo "═══ UPGRADE — $OLD_IMAGE -> $NEW_IMAGE"
+  set_ports 25055 28502 28080
+  local DD; DD=$(mktemp -d /tmp/onrel-upg-XXXX)
+  echo "-- phase 1: boot $OLD_IMAGE and seed"
+  compose_up "$OLD_IMAGE" "$DD" onrelupg || { compose_down onrelupg "$DD"; return 1; }
+  IMG=$(docker inspect onrelupg-app-1 --format '{{.Config.Image}}' 2>/dev/null)
+  check "phase 1 runs the old image" "$OLD_IMAGE" "$IMG"
+  if wait_api; then ok "old image up"; else
+    bad "old image did not come up"; compose_down onrelupg "$DD"; return 1
+  fi
+  seed_and_verify
+  echo "-- phase 2: swap to the new image on the SAME volume"
+  compose_env unused /tmp/unused onrelupg stop app >/dev/null 2>&1
+  compose_env unused /tmp/unused onrelupg rm -f app >/dev/null 2>&1
+  compose_up "$NEW_IMAGE" "$DD" onrelupg || { compose_down onrelupg "$DD"; return 1; }
+  IMG=$(docker inspect onrelupg-app-1 --format '{{.Config.Image}}' 2>/dev/null)
+  check "phase 2 runs the new image" "$NEW_IMAGE" "$IMG"
+  if wait_api; then ok "new image up on old data"; else
+    bad "new image did not come up on old data"; docker logs onrelupg-app-1 2>&1 | tail -20
+    compose_down onrelupg "$DD"; return 1
+  fi
+  FOUND=$(curl -s "$API/api/notebooks" | python3 -c "
+import json,sys
+nbs=json.load(sys.stdin)
+print('yes' if any(n.get('name')=='release-probe' for n in nbs) else 'no')" 2>/dev/null)
+  check "notebook seeded on old image survived the upgrade" "yes" "$FOUND"
+  S=$(curl -s -o /dev/null -w '%{http_code}' "$API/api/sources?sort_by=title&limit=2")
+  check "sort_by=title after upgrade" "200" "$S"
+  echo "  phase 2 migration log:"
+  docker logs onrelupg-app-1 2>&1 | grep -i "migrat" | tail -5 | sed 's/^/    /'
+  compose_down onrelupg "$DD"
+  echo
+}
+
+case "${1:-all}" in
+  fresh)   fresh_test ;;
+  upgrade) upgrade_test ;;
+  all)     fresh_test; upgrade_test ;;
+esac
+
+echo "═══ RESULT: $PASS passed, $FAIL failed"
+[ "$FAIL" -eq 0 ]