Prepare environment is pure infrastructure (Talos image download, sandbox
VMs boot, network setup). Failures here are mostly noisy-runner /
transient infra hiccups (image-download stalls, NIC negotiation, etc.) —
not cozystack code under test. Retry-on-failure is the right policy for
infra setup.
Install Cozystack and Run E2E tests remain single-attempt: they exercise
cozystack code, where retries hide real bugs.
Refines the previous two retry-removal commits.
Signed-off-by: Myasnikov Daniil <myasnikovdaniil2001@gmail.com>
A failed cluster bootstrap or platform install is almost always a real
bug (Talos boot, chart, operator) — not a flake. The 3x retry on these
steps was the direct cause of the 250-min outlier run (24931612182):
each retry compounds 25-30 min of work, and the failure mode persists
across attempts. Single attempt + the existing 'collect-report' step
on always() gives faster signal and a debug archive.
Signed-off-by: Myasnikov Daniil <myasnikovdaniil2001@gmail.com>
Across 5 sampled failure runs (30 PR runs total), 25/25 retry attempts
failed. The retry loop never recovered a flake — it only stretched
deterministic failures (e.g. kubernetes-previous: 11:37 + 7:49 + 10:16
= 29:42 wasted on one broken test). Replace with a single attempt plus
inline diagnostics (HR list + recent events) so the actual failure
surfaces immediately. Re-runs remain available via the standard
'gh run rerun' / empty-commit retry path.
Signed-off-by: Myasnikov Daniil <myasnikovdaniil2001@gmail.com>
Address review feedback from myasnikovdaniil on .github/workflows/pr-labeler.yaml:160,173:
emit core.warning when a Conventional Commits type has no kind/*
mapping or a scope has no area/* mapping. Without the warning, typos
(e.g., "hotfix" instead of "fix") and recurring new scopes silently
fall through to area/uncategorized, masking that the mapping has
drifted.
Assisted-By: Claude <noreply@anthropic.com>
Signed-off-by: Aleksei Sviridkin <f@lex.la>
Address review feedback from myasnikovdaniil on .github/workflows/pr-labeler.yaml:155:
Conventional Commits 1.0 spec item 16 treats BREAKING CHANGE: and
BREAKING-CHANGE: as synonymous footers. The hyphen form was silently
ignored before, so PRs that use it would miss kind/breaking-change.
https://www.conventionalcommits.org/en/v1.0.0/#specification
Assisted-By: Claude <noreply@anthropic.com>
Signed-off-by: Aleksei Sviridkin <f@lex.la>
Address review feedback from myasnikovdaniil on .github/workflows/pr-labeler.yaml:129:
defensive (pr.labels || []) avoids TypeError if the webhook payload
arrives without the labels field on edge cases like stripped edited
events.
Assisted-By: Claude <noreply@anthropic.com>
Signed-off-by: Aleksei Sviridkin <f@lex.la>
Add .github/labels.yml as the canonical label set, synced into the
repository by .github/workflows/labels.yaml using EndBug/label-sync.
Conventions follow the Kubernetes scheme:
https://github.com/kubernetes/test-infra/blob/master/label_sync/labels.md
Six namespaced groups: kind/, priority/, triage/, lifecycle/, area/,
do-not-merge/. Cozystack-specific labels preserved (epic, community,
security/*, size:*).
Migration via aliases keeps references on existing issues and PRs:
- bug -> kind/bug
- enhancement -> kind/feature
- documentation -> kind/documentation
- question -> kind/support
- frozen -> lifecycle/frozen
- stale -> lifecycle/stale
- do-not-merge -> do-not-merge/work-in-progress
delete-other-labels is false on the initial rollout; redundant labels
("do not merge", duplicate, invalid, wontfix) stay until a follow-up
PR removes them after stabilisation.
The labels workflow has a validate job (python3 schema check) that
runs on PR. Sync runs only on push to main, weekly cron, and manual
dispatch. Schema invariants:
- description <= 100 chars (GitHub REST API limit)
- color is 6-char hex without leading #
- unique top-level names
- aliases do not collide with top-level names
PR auto-labeling (.github/workflows/pr-labeler.yaml):
- Parses PR title as Conventional Commits header (type, scope, !).
- type -> kind/* (feat -> kind/feature, fix -> kind/bug, docs ->
kind/documentation, chore/refactor -> kind/cleanup; style, perf,
test, build, ci, revert -> no kind label).
- scope -> area/* via embedded mapping; composite scopes split on
comma. Bracket-style fallback ([scope] description) maps area/*
but cannot infer kind/*.
- '[Backport release-1.x]' prefix is stripped; area/release and
backport labels are added.
- '!' after type or 'BREAKING CHANGE:' footer in body adds
kind/breaking-change.
- Unmapped scope or non-conventional title adds area/uncategorized
to flag for human review.
- Additive only — never removes existing labels.
Hardcoded label references updated:
- .github/ISSUE_TEMPLATE/bug_report.md (bug -> kind/bug)
- .github/workflows/tags.yaml (documentation -> kind/documentation)
AGENTS.md gains an Activation entry pointing agents to labels.yml
as the source of truth and to contributing.md for the title
auto-labeling table.
Assisted-By: Claude <noreply@anthropic.com>
Signed-off-by: Aleksei Sviridkin <f@lex.la>
Review feedback on PR #2460: the Generate changelog using AI step ran
Copilot with --allow-all-tools and GH_TOKEN set to the write-capable
installation token issued to the job (contents: write,
pull-requests: write on all cozystack/* repos). The scope rules in
docs/agents/changelog.md and the step prompt tell the agent not to
use those permissions, but nothing at the token layer prevented it.
Mint a second, read-only installation token from the same app
(same COZYSTACK_CI_APP_ID / COZYSTACK_CI_PRIVATE_KEY, scoped to
contents/pull-requests/metadata read) and pass that one to the AI
step instead. The write-capable token is still used by the checkout,
commit/push, and PR-creation steps that actually need it.
This is defense in depth: even if a future prompt change or agent
misbehavior ignored the scope rules, the token itself has no write
capability on any repository in the cozystack org. No new secret,
no new GitHub App install, no admin-side change — the RO token is
minted in the same workflow from the same app credentials.
Signed-off-by: Myasnikov Daniil <myasnikovdaniil2001@gmail.com>
Review feedback on PR #2460: the existing `[ -f ]` check catches
missing files but a zero-byte `docs/changelogs/v${VERSION}.md` would
still be staged and committed — `git add` + `git commit -s` on a new
empty file succeeds and produces a real commit, leaving the
downstream PR with no actual changelog content.
Add a `[ -s ]` guard after the existence check: if the Generate
changelog using AI step produces an empty file, emit a matching
`::error::` annotation and exit 1 before snapshotting.
Signed-off-by: Myasnikov Daniil <myasnikovdaniil2001@gmail.com>
Three changes to the generate-changelog job to fix the v1.3.0
release pipeline failure (run 24765377017) and make the job robust
to whatever state the Copilot step leaves behind.
1. Add `timeout-minutes: 30` to the Generate changelog using AI
step. On the v1.3.0 re-run the step hung silently for 10+
minutes; with no timeout a hung Copilot would hold a self-hosted
runner for up to 6 hours (job default). The previous successful
run took ~26 minutes, so 30 is a reasonable ceiling.
2. Replace the terse, ambiguous Copilot prompt with a one-liner
that invokes docs/agents/changelog.md directly. The "Scope and
boundaries" section added to that doc in the previous commit is
now the single source of truth for what the agent may and may
not do, so the workflow only needs to pass the version and
point at the relevant doc. VERSION is moved to step env: to
match GitHub's workflow-injection hardening guidance.
3. Rewrite the Create changelog branch and commit step:
- add `set -euo pipefail` so any failure is visible
- validate the file exists up front and fail loud with
`::error::` if not
- copy the file to a tempfile BEFORE `git checkout -b`, so the
checkout to `origin/main` cannot remove it (this is the fix
for the original pathspec error the v1.3.0 run hit)
- use `trap` to clean up the tempfile on any exit path
- move VERSION to env
- drop the dead "no changes to commit" branch: the check_changelog
step earlier in the job gates this step on the file being
absent from origin/main, so `git add` + `git commit` must
produce a diff. If they don't (e.g. Copilot emitted an empty
file), fail loud instead of pushing an empty branch.
Signed-off-by: Myasnikov Daniil <myasnikovdaniil2001@gmail.com>
The website repo is switching to a permanent `content/en/docs/next/`
trunk (cozystack/website#495). Released version directories are no
longer pre-created — the upstream release workflow is now responsible
for explicitly promoting `next/` → `vX.Y/` on new minor/major releases
via `make release-next`.
Update the `update-website-docs` job to match that contract:
- New `Determine if this release promotes next/` step parses the tag
and only enables promotion for final `vX.Y.Z` tags where
`content/en/docs/vX.Y/` does not already exist. Prereleases and
patch releases keep the old behaviour (target routing is handled
by the website Makefile on its own).
- New `Promote next/ to released version` step runs
`make release-next RELEASE_TAG=...` before `make update-all`, so
the subsequent `update-all` routes into the freshly-created
`vX.Y/` directory and `next/` stays untouched as the trunk for
the following release.
- `git add content hugo.yaml` to pick up the version registration
that `release-next` writes via `register_version.sh`.
- Convert the existing `update-all` step to use `env:` vars,
matching the new step and addressing the workflow-injection
hardening guidance.
Signed-off-by: Myasnikov Daniil <myasnikovdaniil2001@gmail.com>
The cozystack-ci[bot] noreply email with brackets is rejected by DCO
probot as an invalid email address, causing release PRs to fail DCO
checks.
Co-Authored-By: Claude <noreply@anthropic.com>
Signed-off-by: Andrei Kvapil <kvapss@gmail.com>
The GH_PAT secret tied to cozystack-bot is no longer valid after
migrating release workflows to the cozystack-ci GitHub App in #2351.
This broke the resolve_assets and e2e jobs for release PRs.
Co-Authored-By: Claude <noreply@anthropic.com>
Signed-off-by: Andrei Kvapil <kvapss@gmail.com>
Replace the cozystack-bot personal access token (GH_PAT) with a
GitHub App (cozystack-ci) installation token across all CI/CD
workflows. This improves security by using short-lived, scoped
tokens instead of a long-lived PAT tied to a user account.
Changes:
- tags.yaml: use app token in all 3 jobs (prepare-release,
generate-changelog, update-website-docs)
- auto-release.yaml: use app token for daily patch releases
- pull-requests-release.yaml: use app token for release finalization
The cozystack-ci GitHub App (ID: 3297617) is installed org-wide
with contents:write and pull-requests:write permissions. Secrets
COZYSTACK_CI_APP_ID and COZYSTACK_CI_PRIVATE_KEY are set at org
level.
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
Always force-create and force-push the API submodule tag so it stays
in sync with the main version tag, even when re-tagging.
Co-Authored-By: Claude <noreply@anthropic.com>
Signed-off-by: Andrei Kvapil <kvapss@gmail.com>
- Remove broken `git push origin HEAD` in detached HEAD state (commit
artifacts are already pushed via the "Create release branch" step)
- Make subtag push idempotent: use explicit refs/tags/ prefix and
tolerate already-existing remote tags after verifying they point to
the correct commit
Co-Authored-By: Claude <noreply@anthropic.com>
Signed-off-by: Andrei Kvapil <kvapss@gmail.com>
## What this PR does
Reduces flakiness in E2E tests by addressing the most common failure
modes.
### Kubernetes tenant tests (`run-kubernetes.sh`)
- Increase node Ready timeout from 2m to 5m — CI runners are shared and
resource-constrained
- Fail fast when nodes are not Ready — saves ~7 minutes per failed
attempt by not running LB/NFS tests that will also fail
- Delete stale Kubernetes resources at test start — retries provision
from scratch instead of patching stuck state
- Wait for port-forward to be ready before using it (fixes race
condition)
- Reduce version check polling interval from 5s to 1s
### All app tests (postgres, redis, kafka, clickhouse, mariadb, mongodb,
qdrant, foundationdb, openbao, vminstance, bucket)
- Add pre-cleanup of stale resources from previous failed retries so
each attempt starts clean
### Test runner (`cozytest.sh`)
- Clean up temp directory on test failure (was only cleaned on success,
leaking `/tmp` dirs)
### Release note
```release-note
NONE
```
<!-- This is an auto-generated comment: release notes by coderabbit.ai
-->
## Summary by CodeRabbit
* **Tests**
* Improved e2e robustness by deleting stale resources before creation
(ignore-if-missing, bounded time) and killing leftover port-forwards.
* Added an exit cleanup to consistently remove temporary artifacts and
teardown port-forwards on any exit.
* Added readiness polling for forwarded endpoints (curl with timeout);
made API/version retries more responsive.
* Extended node readiness wait (2m → 5m), dump debug info and fail fast
if unready.
* Made load‑balancer reachability checking explicit and more reliable.
<!-- end of auto-generated comment: release notes by coderabbit.ai -->
Without explicit timeouts, stuck jobs use the GitHub Actions
default of 360 minutes (6 hours). This blocks the concurrency
group and prevents new runs from starting.
Set reasonable limits:
- Build: 30 minutes (normally ~5 min)
- E2E: 120 minutes (normally ~60 min)
Assisted-By: Claude <noreply@anthropic.com>
Signed-off-by: Aleksei Sviridkin <f@lex.la>
The pull-requests workflow used paths-ignore at the trigger level, which
prevented the entire workflow from running on docs-only PRs. This meant
the required "E2E Tests" check was never created, blocking merge for
non-admin users.
Replace trigger-level paths-ignore with a detect-changes job using
dorny/paths-filter. The workflow now always triggers (so checks are
always reported), but build and downstream jobs are skipped when only
docs files change.
Signed-off-by: IvanHunters <xorokhotnikov@gmail.com>
Replace pre-rendered static YAML application with direct helm chart
installation in e2e tests. The chart directory with correct values is
already present in the sandbox after pr.patch application.
- Remove CRD/operator artifact upload/download from CI workflow
- Remove copy-installer-manifest target from testing Makefile
- Use helm upgrade --install from local chart in e2e-install-cozystack.bats
Co-Authored-By: Claude <noreply@anthropic.com>
Signed-off-by: Aleksei Sviridkin <f@lex.la>
Add -talos suffix to the default variant output file for consistency
with -generic and -hosted variants. Update all references in CI
workflows, e2e tests, upload scripts, and testing Makefile.
Co-Authored-By: Claude <noreply@anthropic.com>
Signed-off-by: Andrei Kvapil <kvapss@gmail.com>
## What this PR does
This patch adds a conditional for running on a statically defined VM the
maintainers have SSH access to if the pull request has a `debug` label.
This is useful for debugging failing workflows when the diagnostic info
from the pipeline is insufficient.
### Release note
```release-note
[ci] Run builds on a static VM with SSH access if the PR has a debug
label.
```
Signed-off-by: Timofei Larkin <lllamnyp@gmail.com>
## What this PR does
Use GitHub Copilot CLI for automatic changelog generation on tagged
releases.
### How it works
When a new version tag is pushed, the `generate-changelog` job in
`tags.yaml`:
1. Checks out the `main` branch with full history and tags
2. Verifies that a changelog file doesn't already exist in
`docs/changelogs/`
3. Installs GitHub Copilot CLI (`@github/copilot` npm package)
4. Runs Copilot CLI in non-interactive mode (`-p` flag) with
`--allow-all-tools --allow-all-paths` to generate the changelog
following the instructions in `docs/agents/changelog.md`
5. Commits the generated file to a `changelog-vX.Y.Z` branch and opens a
PR to `main`
### Authentication
- `COPILOT_GITHUB_TOKEN` secret — fine-grained PAT with **"Copilot
Requests: Read"** account permission, used to authenticate Copilot CLI
- `GH_PAT` secret — used by `gh` CLI inside the Copilot session to query
PR authors via GitHub API
### Release note
```release-note
[ci] Replaced Gemini with GitHub Copilot CLI for automatic changelog generation on release tags
```
- Add pattern rule for building cozypkg binaries per platform
(assets-cozypkg-<os>-<arch>) with checksums generation
- Add Version variable to cozypkg CLI injected via ldflags
- Split manifests into separate cozystack-crds.yaml and
cozystack-operator.yaml files
- Update CI workflow to handle CRDs and operator as separate artifacts
- Update e2e tests and upload script for new manifest structure
- Suppress git describe stderr when no tags exist
Co-Authored-By: Claude <noreply@anthropic.com>
Signed-off-by: Andrei Kvapil <kvapss@gmail.com>
Replace unreliable getLatestRelease() API with semver-based max version
detection. After publishing a backport release, explicitly restore the
latest flag on the highest semver release to handle cases where GitHub
API ignores make_latest: 'false'.
Also remove dead code (unused steps) from tags.yaml workflow.
Co-Authored-By: Claude <noreply@anthropic.com>
Signed-off-by: Andrei Kvapil <kvapss@gmail.com>
Allows maintainers to trigger test rerun by commenting /retest on a PR.
The workflow finds the latest run for the PR and reruns starting from
the "Prepare environment" job, skipping the build step.
Co-Authored-By: Claude <noreply@anthropic.com>
Signed-off-by: Andrei Kvapil <kvapss@gmail.com>
actions/checkout configures http.extraheader with GITHUB_TOKEN which
takes priority over URL credentials. This caused tag pushes to
authenticate as github-actions instead of cozystack-bot, preventing
the Versioned Tag workflow from being triggered.
Co-Authored-By: Claude <noreply@anthropic.com>
Signed-off-by: Andrei Kvapil <kvapss@gmail.com>
This PR adds a new GitHub workflow that automatically creates patch
releases for maintenance branches.
## What it does
- Runs daily at 2:00 AM CET (1:00 UTC)
- Checks all `release-X.Y` branches
- For each branch, finds the latest published release with tag `vX.Y.Z`
(without suffixes like -rc, -alpha, -beta)
- If new commits exist after the release, creates a new tag `vX.Y.Z+1`
and force-pushes it
- This triggers the existing tag workflow to build and create a release
PR
## Why releases instead of tags
The workflow checks published releases (not just tags) because if a
release PR hasn't been merged yet, the workflow should run again the
next day and move the tag to newer commits if they exist.
<!-- This is an auto-generated comment: release notes by coderabbit.ai
-->
## Summary by CodeRabbit
* **Chores**
* Automated patch release workflow configured for release branches,
enabling automatic version tagging when new commits are detected.
<sub>✏️ Tip: You can customize this high-level summary in your review
settings.</sub>
<!-- end of auto-generated comment: release notes by coderabbit.ai -->
This PR adds a GitHub Actions workflow that automatically updates
release notes from changelog files when changes are merged to main.
The workflow:
- Triggers on push to main branch
- Processes up to 30 releases from the first page
- For each release, checks if a corresponding changelog file exists in
`docs/changelogs/`
- Updates the release notes with the changelog content if it exists and
differs from the current content
- Skips releases that are already up to date to avoid unnecessary API
calls
This automates the process of keeping release notes synchronized with
changelog files.
<!-- This is an auto-generated comment: release notes by coderabbit.ai
-->
## Summary by CodeRabbit
* **Chores**
* Automated workflow added to synchronize GitHub releases with local
changelog files, updating release notes only when content differs.
* **Documentation**
* Changelogs and release notes reorganized and expanded with improved
sectioning and new tooling/documentation entries.
* **New Features**
* Added VM disk SVG icon entry under Features/Improvements.
* **Bug Fixes**
* Pinned CoreDNS image tag for reproducible deployments.
* Fixed a documentation typo that prevented a component from displaying
in the web UI.
<sub>✏️ Tip: You can customize this high-level summary in your review
settings.</sub>
<!-- end of auto-generated comment: release notes by coderabbit.ai -->
## What this PR does
The cozy-lib library package got complicated enough to warrant its own
unit tests. Since unit tests are a "good thing" (tm), a somewhat generic
framework for running all kinds of unit tests was introduced into the CI
pipeline and Makefile targets. For now all it runs is `make test`
against the `packages/{library,apps,system,extra}/*` directories,
wherever a `test` target is present in the Makefile, and for now this is
only for the `cozy-lib` Helm library chart.
### Release note
```release-note
[ci,dx] Introduce a scaffold for running unit tests locally and in CI
and add the first unit tests for the cozy-lib helper Helm chart.
```
Signed-off-by: Timofei Larkin <lllamnyp@gmail.com>
This patch sources the REGISTRY env var from GitHub actions variables
instead of secrets, so pull requests from forked repos work correctly.
```release-note
[ci] Source the REGISTRY env var from actions' variables, not secrets,
so external pull requests can work.
```
Signed-off-by: Timofei Larkin <lllamnyp@gmail.com>
