- Log .status.history regardless of content so a silently empty result
(Flux field rename) shows up in CI logs, and treat empty history on
a Ready HelmRelease as a distinct failure. A Ready HR by definition
has at least one snapshot; anything else is a shape-drift signal.
- Replace the unquoted heredoc in remediation-guard.sh with a printf |
grep pipeline. printf %s treats statuses as literal payload (no $
expansion surprises for future callers), grep --quiet --extended-regexp
returns exit status the caller can forward directly.
- Share the etcd-absent values file between both invariant tests
(packages/apps/kubernetes/tests/values-ci-no-etcd.yaml) instead of
duplicating the --set block.
- Fix typo "override applied" -> "override is applied" in the
Kubernetes ApplicationDefinition.
- Add a coupling comment in the ApplicationDefinition annotation that
points at the wait-for-kubeconfig init deadline in _helpers.tpl, so
a future operator raising the HR timeout updates the init deadline
too.
- Clarify the per-annotation timeout comment in rest.go so it stops
implying the feature is Kubernetes-only (it is not - only today's
one user is).
Assisted-By: Claude <noreply@anthropic.com>
Signed-off-by: Aleksei Sviridkin <f@lex.la>
The image-busybox Makefile target + images/busybox/Dockerfile wrapper
just rebuilt an upstream busybox digest as ghcr.io/cozystack/cozystack/
busybox. Payload is a one-shot sh loop run once per pod; the pinned
upstream digest is already immutable, so maintaining a private mirror
adds churn (rebuild on every release) for no real hardening benefit.
Delete the wrapper and reference docker.io/library/busybox:<digest>
directly from images/busybox.tag. Document the choice in _helpers.tpl.
Also drop the false coupling in the go table test: the "unrelated
kind without configured timeout" case used the real kind name Qdrant,
which tied the test to the Qdrant ApplicationDefinition for no reason.
Switch to a clearly fictional kind so a future Qdrant timeout override
does not break this assertion.
Assisted-By: Claude <noreply@anthropic.com>
Signed-off-by: Aleksei Sviridkin <f@lex.la>
Replace the hardcoded r.kindName == "Kubernetes" switch in rest.go
with a config-driven path. The ApplicationDefinition CR now accepts a
release.cozystack.io/helm-install-timeout annotation that is parsed
at cozystack-api startup into config.ReleaseConfig.HelmInstallTimeout
and applied to both Install.Timeout and Upgrade.Timeout on the
rendered HelmRelease. Applications that leave the annotation unset
keep flux defaults so their failed installs remediate on the normal
cadence - only the Kubernetes kind carries the override and gets a
15m budget. New kinds with a similar race can opt in by setting the
same annotation; no rest.go patch needed.
Kubernetes-rd sets the annotation to 15m. Table-driven test in
rest_timeout_test.go covers three cases: Kubernetes with 15m, Qdrant
unset, and an arbitrary future kind with 20m - all of which pin the
Remediation.Retries == -1 contract as well.
Assisted-By: Claude <noreply@anthropic.com>
Signed-off-by: Aleksei Sviridkin <f@lex.la>
The previous change set Install.Timeout and Upgrade.Timeout to 15m
on every Application's parent HelmRelease, but the admin-kubeconfig
race documented in #2412 is specific to the Kubernetes kind: only its
parent chart creates CAPI/Kamaji resources whose admin-kubeconfig
Secret is asynchronously provisioned and mounted by Deployments in
the same chart. Other kinds (Qdrant, MongoDB, Postgres, ...) have no
such race and should not have their failed installs linger three times
longer before flux triggers remediation.
Gate the timeout on r.kindName == "Kubernetes". Rewrite
rest_timeout_test.go to cover both sides: Kubernetes must get a
>= 15m timeout, other kinds must keep the flux defaults. Both tests
also pin Install/Upgrade Remediation.Retries == -1 so a future edit
that removes unbounded remediation would show up as a red test.
Assisted-By: Claude <noreply@anthropic.com>
Signed-off-by: Aleksei Sviridkin <f@lex.la>
Parent HelmRelease created by cozystack-api for Kubernetes tenants
contains CAPI/Kamaji resources (Cluster, KamajiControlPlane,
MachineDeployment) that asynchronously provision the
*-admin-kubeconfig Secret. Three Deployments in the same chart
(cluster-autoscaler, kccm, kcsi-controller) mount that Secret
directly, so the helm-wait cannot complete until control-plane
bootstrap finishes.
Default flux helm-controller timeout is too short for a cold-node
first-tenant bootstrap (image pull + etcd bootstrap + apiserver Ready
+ admin-kubeconfig generation routinely exceed it). On timeout,
install.remediation triggers uninstall, which removes the Cluster CR
and restarts the cycle indefinitely.
Bumping Install.Timeout and Upgrade.Timeout to 15m gives realistic
bootstrap headroom while remaining bounded.
Assisted-By: Claude <noreply@anthropic.com>
Signed-off-by: Aleksei Sviridkin <f@lex.la>
Adds a failing unit test for convertApplicationToHelmRelease asserting
that Install.Timeout and Upgrade.Timeout are at least 15 minutes. The
default flux helm-controller timeout is too short to cover cold-start
Kamaji control-plane bootstrap (image pull + etcd bootstrap + apiserver
Ready + admin-kubeconfig Secret generation) and causes install
remediation loops.
Assisted-By: Claude <noreply@anthropic.com>
Signed-off-by: Aleksei Sviridkin <f@lex.la>
Tenant applications live in a child namespace computed from the Tenant
name, not in the HelmRelease's namespace. Scope the WorkloadMonitor
watch cluster-wide when kindName is Tenant and reverse-map WM events
back to the owning HelmRelease via computeTenantNamespace. The
condition-enrichment path now looks up monitors in the computed child
namespace too, so WorkloadsReady is populated for Tenants.
Buffer WorkloadMonitor events that arrive before the initial-events-end
bookmark and replay them once the bookmark is emitted, to preserve the
watch-list contract while not dropping workload-state transitions that
happen during the snapshot window.
Pass the fresh WorkloadMonitor object from the watch event into the
conversion path so WorkloadsReady reflects the state that triggered
the event, even when the cache client is lagging behind the watch
client.
Derive WorkloadsReady.LastTransitionTime from a stable source
(max of HelmRelease creation and condition timestamps, plus monitor
timestamps) instead of metav1.Now(), so repeated conversions of the
same underlying state produce identical timestamps.
Assisted-By: Claude <noreply@anthropic.com>
Signed-off-by: Aleksei Sviridkin <f@lex.la>
Add two features to improve application observability in the dashboard:
1. WorkloadsReady condition on Application status
- Query WorkloadMonitor resources to determine if all pods are running
- Expose WorkloadsReady as a separate condition alongside Ready
- Ready continues to reflect HelmRelease state only (no override) to
preserve backward compatibility with tooling and avoid false-negatives
during normal startup windows when pods are still coming up
- Handle three states: operational, not operational, unknown (pending)
- Fail-open on WorkloadMonitor query errors
- Integrate with Application Watch to emit MODIFIED events on
WorkloadMonitor changes (with initial-events-end safety)
- Register cozystack.io/v1alpha1 types in API server scheme
2. Events tab in dashboard
- Show Kubernetes Events scoped to application namespace
- Use status.namespace for Tenant applications
- Include both lastTimestamp and eventTime columns for K8s compat
3. Bug fix: WorkloadMonitor Operational status persistence
- Operational was written to stale 'monitor' instead of 'fresh'
inside RetryOnConflict, so it was never persisted to the cluster
Closes#2359Closes#2360
Assisted-By: Claude <noreply@anthropic.com>
Signed-off-by: Aleksei Sviridkin <f@lex.la>
Replace bare "Tenant" string literals with the validation.TenantKind constant in all test table entries, struct fields, and TypeMeta. Prevents silent drift if the canonical kind string is ever renamed.
Assisted-By: Claude <noreply@anthropic.com>
Signed-off-by: Aleksei Sviridkin <f@lex.la>
The convertHelmReleaseToApplication fall-through that populates Status.Namespace for tenant resources still compared r.kindName against the bare string "Tenant". The validation package already exposes TenantKind for exactly this reason: if the kind string ever diverges from the ApplicationDefinition source of truth, the constant and the call site drift together rather than silently desynchronizing.
Add TestConvertHelmReleaseToApplication_TenantNamespaceKindGate to pin both sides of the gate: tenant kind populates Status.Namespace, non-tenant kind leaves it empty. Also rewrite the TestUpdate comment to describe the upsert invariant directly instead of referencing an issue number.
Assisted-By: Claude <noreply@anthropic.com>
Signed-off-by: Aleksei Sviridkin <f@lex.la>
Four follow-ups from review round 4:
1. BLOCKER: the Update(forceAllowCreate=true) path delegates to Create() when the object does not yet exist (rest.go:452) — the typical kubectl apply upsert flow. Add TestUpdate_ForceAllowCreate_RejectsTenantDashName using a fake client so a future refactor of that delegation cannot silently bypass the tenant name check that r.validateNameFormat alone cannot catch.
2. BLOCKER: the e2e BATS test used || true inside the command substitution, which swallowed the kubectl exit code. Rework the test to capture exit code and stdout+stderr explicitly, then assert the exit code is non-zero before asserting on the error message. This distinguishes validation-success (kubectl exit 0 — regression) from environmental failures (exit non-zero but wrong message) from the happy path.
3. Extract TenantKind = "Tenant" as a named constant in the validation package with a comment pointing at the upstream ApplicationDefinition source of truth, and switch the kindName check to use it.
4. Add a clarifying comment on TestValidateApplicationName_TenantLengthFallthrough that it pins an architectural layering decision and is not a user-facing requirement, so a future promotion of tenant length into tenant-specific wording is a legitimate change rather than a test regression.
Assisted-By: Claude <noreply@anthropic.com>
Signed-off-by: Aleksei Sviridkin <f@lex.la>
ValidateApplicationName previously delegated to IsDNS1035Label, which permits hyphens. The tenant Helm chart's tenant.name helper rejects them at template time, so users saw a successful kubectl apply followed by a Flux reconciliation error.
Extend ValidateApplicationName with a kindName parameter and enforce alphanumeric-only names for the Tenant kind, matching the documented naming rules. Wire the check into REST.Create via a small validateNameFormat wrapper that mirrors validateNameLength.
Assisted-By: Claude <noreply@anthropic.com>
Signed-off-by: Aleksei Sviridkin <f@lex.la>
Reject Tenant creation when the computed workload namespace (parent
namespace + "-" + tenant name) would exceed the 63-character DNS-1123
label limit. Previously only the per-name length was checked against
the Helm release limit, so a deeply-nested tenant could pass Create()
and then fail later as an opaque HelmRelease reconcile error.
Assisted-By: Claude <noreply@anthropic.com>
Signed-off-by: Aleksei Sviridkin <f@lex.la>
Root-host validation for Tenant names is no longer needed here.
The underlying issue (namespace.cozystack.io/host label exceeding
63-char limit) will be addressed in #2002 by moving the label
to an annotation.
Name length validation now only checks the Helm release name
limit (53 - prefix length), which applies uniformly to all
application types.
Co-Authored-By: Claude <noreply@anthropic.com>
Signed-off-by: Aleksei Sviridkin <f@lex.la>
- Fix goimports order: duration before validation/field
- Show rootHost in error messages only for Tenant kind where it
actually affects the length calculation
Co-Authored-By: Claude <noreply@anthropic.com>
Signed-off-by: Aleksei Sviridkin <f@lex.la>
- Return field.ErrorList from validateNameLength for consistent
apierrors.NewInvalid error shape (was NewBadRequest)
- Add klog warning when YAML parsing fails in parseRootHostFromSecret
- Fix maxHelmReleaseName comment to accurately describe Helm convention
- Add note that root-host changes require API server restart
- Replace interface{} with any throughout openapi.go and rest.go
- Remove trailing blank line in const block
Co-Authored-By: Claude <noreply@anthropic.com>
Signed-off-by: Aleksei Sviridkin <f@lex.la>
Skip DNS-1035 and length validation on Update since Kubernetes names
are immutable — validating would block updates to pre-existing resources
with non-conforming names. Replace fmt.Printf with klog for structured
logging consistency.
Co-Authored-By: Claude <noreply@anthropic.com>
Signed-off-by: Aleksei Sviridkin <f@lex.la>
- Add DNS-1035 format validation to Update path (was only in Create)
- Simplify Secret reading by reusing existing scheme instead of
creating a separate client
- Add nil secret test case for parseRootHostFromSecret
Co-Authored-By: Claude <noreply@anthropic.com>
Signed-off-by: Aleksei Sviridkin <f@lex.la>
Add protection against negative or zero maxLen when release prefix or
root host are too long, returning a clear configuration error instead of
a confusing "name too long" message. Add corresponding test cases.
Co-Authored-By: Claude <noreply@anthropic.com>
Signed-off-by: Aleksei Sviridkin <f@lex.la>
Read root-host from cozystack-values secret at API server startup
and use it to compute maximum allowed name length for applications.
For all apps: validates prefix + name fits within the Helm release
name limit (53 chars). For Tenants: additionally checks that the
host label (name + "." + rootHost) fits within the Kubernetes label
value limit (63 chars).
This replaces the static 40-char limit with a dynamic calculation
that accounts for the actual cluster root host length.
Ref: https://github.com/cozystack/cozystack/issues/2001
Co-Authored-By: Claude <noreply@anthropic.com>
Signed-off-by: Aleksei Sviridkin <f@lex.la>
Add validation to ensure Application names (including Tenants) conform
to DNS-1035 format. This prevents creation of resources with names
starting with digits, which would cause Kubernetes resource creation
failures (e.g., Services, Namespaces).
DNS-1035 requires names to:
- Start with a lowercase letter [a-z]
- Contain only lowercase alphanumeric or hyphens [-a-z0-9]
- End with an alphanumeric character [a-z0-9]
Also fixes broken validation.go that referenced non-existent internal
types (apps.Application, apps.ApplicationSpec).
Fixes: https://github.com/cozystack/cozystack/issues/1538
Co-Authored-By: Claude <noreply@anthropic.com>
Signed-off-by: Aleksei Sviridkin <f@lex.la>
- Add resourceVersion handling for Watch requests by filtering ADDED events
based on the resourceVersion provided by the client
- Forward bookmark events from underlying HelmRelease watchers to clients
for proper resourceVersion synchronization
- Extract MaxResourceVersion helper using meta.EachListItem for cleaner code
- This ensures clients don't receive duplicate objects they already have
from List+Watch patterns
Co-Authored-By: Claude <noreply@anthropic.com>
Signed-off-by: Andrei Kvapil <kvapss@gmail.com>
Replace the chart field with chartRef for referencing Helm charts via
ExternalArtifact resources. This enables the Package controller to
manage chart sources centrally.
Changes:
- Add chartRef field to CozystackResourceDefinition spec
- Remove chart field (deprecated)
- Remove validation (moved to controller)
- Update lineage mapper for new field structure
- Regenerate openapi specs
Co-Authored-By: Claude <noreply@anthropic.com>
Signed-off-by: Andrei Kvapil <kvapss@gmail.com>
Fix getVersion to parse "0.1.4+abcdef" format (with "+" separator)
instead of incorrectly looking for "sha256:" prefix.
Co-Authored-By: Claude <noreply@anthropic.com>
Signed-off-by: Andrei Kvapil <kvapss@gmail.com>
Controller-runtime cache doesn't support field selectors, causing
incorrect filtering when using kubectl with field selectors like
--field-selector=metadata.namespace=tenant-kvaps or metadata.name=test.
Changes:
- Created pkg/registry/fields package with ParseFieldSelector utility
- Refactored field selector parsing logic in application, tenantmodule,
and tenantsecret registries to use common implementation
- Implemented manual filtering for metadata.name and metadata.namespace
in List() and Watch() methods
- Removed Raw field usage and field selectors from client.ListOptions
- Label selectors passed directly via LabelSelector field
Field selectors now properly filter resources by name and namespace
through manual post-processing after label-based filtering.
See: https://github.com/kubernetes-sigs/controller-runtime/issues/612
Co-Authored-By: Claude <noreply@anthropic.com>
Signed-off-by: Andrei Kvapil <kvapss@gmail.com>
Replace Helm lookup functions with FluxCD valuesFrom mechanism for
reading cluster and namespace configuration.
Changes:
- Create Secret cozystack-values in each namespace with values.yaml key
containing _cluster and _namespace configuration as nested YAML
- Configure HelmReleases to read from this Secret via valuesFrom
(valuesKey defaults to values.yaml, so it can be omitted)
- Update cozy-lib helpers to access config via .Values._cluster
- Add default values for required _cluster keys to ensure all fields exist
- Update Go code (cozystack-api and helm reconciler) to use new format
This eliminates the need for Helm lookup functions while maintaining
the same configuration interface for charts.
Co-Authored-By: Claude <noreply@anthropic.com>
Signed-off-by: Andrei Kvapil <kvapss@gmail.com>
- Simplify ByName with strings.Compare
- Avoid string allocation in ByNamespacedName by comparing fields separately
Co-Authored-By: Claude <noreply@anthropic.com>
Signed-off-by: Aleksei Sviridkin <f@lex.la>
Add sorting helper package with generic functions for sorting
resources by name (cluster-scoped) or namespace/name (namespace-scoped).
Apply sorting to:
- TenantNamespace (by name)
- TenantSecret (refactored to use helper)
- TenantModule (new)
- Application (new)
Co-Authored-By: Claude <noreply@anthropic.com>
Signed-off-by: Aleksei Sviridkin <f@lex.la>
- Add three application metadata labels to HelmRelease:
- apps.cozystack.io/application.kind
- apps.cozystack.io/application.group
- apps.cozystack.io/application.name
- Replace shouldIncludeHelmRelease filtering with label-based filtering
in Get, List, and Update methods
- Always add kind and group label requirements in List for precise filtering
- Update CozystackResourceDefinitionController to watch only HelmReleases
with cozystack.io/ui=true label
- Update LineageControllerWebhook to extract metadata directly from
HelmRelease labels instead of using mapping configuration
- Add functionality to update HelmRelease chart from CozystackResourceDefinition
using label selectors
Signed-off-by: Andrei Kvapil <kvapss@gmail.com>
This commit refactors the apiserver REST handlers to use typed objects
(appsv1alpha1.Application) instead of unstructured.Unstructured, eliminating
the need for runtime conversions and simplifying the codebase.
Additionally, it fixes an issue where UnstructuredList objects were using
the first registered kind from typeToGVK instead of the kind from the
object's field when multiple kinds are registered with the same Go type.
This is a more comprehensive fix for the problem addressed in
https://github.com/cozystack/cozystack/pull/1630, which was reverted in
https://github.com/cozystack/cozystack/pull/1677.
The fix includes the upstream fix from kubernetes/kubernetes#135537,
which enables short-circuit path for UnstructuredList similar to regular
Unstructured objects, using GVK from the object field instead of
typeToGVK.
Changes:
- Refactored rest.go handlers to use typed Application objects
- Removed unstructured.Unstructured conversions
- Fixed UnstructuredList GVK handling
- Updated dependencies in go.mod/go.sum
- Added e2e test for OpenAPI validation
- Updated Dockerfile
Signed-off-by: Andrei Kvapil <kvapss@gmail.com>
## What this PR does
This patch fixes an issue with the Cozystack API server that causes it
to respond with the first registered dynamic list kind when listing
dynamic resources. E.g., when running `k get postgreses`, the raw JSON
response from the cozystack API would be
```json
{
"apiVersion":"apps.cozystack.io/v1alpha1",
"kind":"BootBoxList",
"items": [
{
"apiVersion":"apps.cozystack.io/v1alpha1",
"kind":"Postgres",
...
},
...
],
...
}
```
The root cause is the way the `Typer` interface is implemented for the
`runtime.Scheme` where the dynamic types are registered. Since the base
type for all dynamic types is a `&cozyv1alpha1.Application{}`, all types
are registered under the same key and the `Typer` defaults to the first
`GroupVersionKind` that was registered. Only when a correctly formed
`&unstructured.Unstructured{}` is returned by the API, is this resolving
logic circumvented and the `GroupVersionKind` is instead inferred from
the fields of the returned object. Even an `UnstructuredList` is not
acceptable as a return type, instead the `items` key should be directly
set on the underlying `map[string]interface{}`.
This patch implements the changes detailed above. Additionally, the
following features, fixes, and improvements are added:
* Makefile targets to build and run the Cozystack API locally, against a
Kubernetes server in the environment's KUBECONFIG. Debugging with
Delve is also supported.
* CI tests are added to verify the new changes.
* A bug in the registration of the corev1alpha1 types is fixed.
* Updated the `ConvertToTable` function to properly handle list kinds
which might be of the `&unstructured.Unstructured{}` concrete type
(not an `UnstructuredList`).
* The scheme used by the API server's Client and WatchClient is
separated from the scheme used to serve dynamic types.
* The client config for reading the runtime configuration now uses the
controller-runtime, which handles flags and environment variables
properly, unlike `clientcmd`.
### Release note
```release-note
[api] Fix incorrect list kind for list requests to the Cozystack API for
dynamic resources. Add Makefile targets for local testing. Minor schema
building improvements.
```
Signed-off-by: Timofei Larkin <lllamnyp@gmail.com>
This patch changes all clients in the Cozystack API server to typed
ones from the controller runtime. This should improve the performance of
the API server and simplifies the code by removing work with
unstructured objects and dynamic clients.
```release-note
[api] Use typed and cache-backed k8s clients in the Cozystack API to
improve performance. Get rid of operations on unstructured objects and
use of dynamic clients.
```
Signed-off-by: Timofei Larkin <lllamnyp@gmail.com>
The Cozystack API server lists TenantNamespaces by running a
SubjectAccessReview against every single requested namespace to see if
the user can create a WorkloadMonitor there. Will this is robust in
terms of permissions, delegating the authorization decision to the k8s
API, this is incredibly inefficient and has caused high latency to the
API. This patch simplifies the logic by instead getting the user's
groups and checking if the namespace contains a rolebinding for that
group. That way listing TenantNamespaces is reduced to a list call to
the k8s API for namespaces and another list call for rolebindings across
all namespaces, while authorization is done on the Cozystack API server
instead of making further calls to the k8s API.
```release-note
[api] Optimize listing of TenantNamespaces, fixes a bug causing very
high latency to the k8s API.
```
Signed-off-by: Timofei Larkin <lllamnyp@gmail.com>
This patch refactors the secret selectors to use the
`internal.cozystack.io/tenantresource` label for managing secret
visibility and removes any selectors based on it or the previous
`apps.cozystack.io/tenantresource` label, the idea being that this label
will only ever be set by the controller.
```
[controller,api] Refactor labels for the secret selector.
```
Signed-off-by: Timofei Larkin <lllamnyp@gmail.com>
[cozystack-controller] Introduce new dashboard-controller
[dashboard] Introduce new dashboard based on openapi-ui
Co-authored-by: kklinch0 <kklinch0@gmail.com>
Signed-off-by: kklinch0 <kklinch0@gmail.com>
Signed-off-by: Andrei Kvapil <kvapss@gmail.com>
