[tests] Stabilize E2E kubernetes tests (#2262)
## What this PR does Reduces flakiness in E2E tests by addressing the most common failure modes. ### Kubernetes tenant tests (`run-kubernetes.sh`) - Increase node Ready timeout from 2m to 5m — CI runners are shared and resource-constrained - Fail fast when nodes are not Ready — saves ~7 minutes per failed attempt by not running LB/NFS tests that will also fail - Delete stale Kubernetes resources at test start — retries provision from scratch instead of patching stuck state - Wait for port-forward to be ready before using it (fixes race condition) - Reduce version check polling interval from 5s to 1s ### All app tests (postgres, redis, kafka, clickhouse, mariadb, mongodb, qdrant, foundationdb, openbao, vminstance, bucket) - Add pre-cleanup of stale resources from previous failed retries so each attempt starts clean ### Test runner (`cozytest.sh`) - Clean up temp directory on test failure (was only cleaned on success, leaking `/tmp` dirs) ### Release note ```release-note NONE ``` <!-- This is an auto-generated comment: release notes by coderabbit.ai --> ## Summary by CodeRabbit * **Tests** * Improved e2e robustness by deleting stale resources before creation (ignore-if-missing, bounded time) and killing leftover port-forwards. * Added an exit cleanup to consistently remove temporary artifacts and teardown port-forwards on any exit. * Added readiness polling for forwarded endpoints (curl with timeout); made API/version retries more responsive. * Extended node readiness wait (2m → 5m), dump debug info and fail fast if unready. * Made load‑balancer reachability checking explicit and more reliable. <!-- end of auto-generated comment: release notes by coderabbit.ai -->
This commit is contained in:
commit
dfe16ed9b4
15 changed files with 58 additions and 22 deletions
2
.github/workflows/pull-requests.yaml
vendored
2
.github/workflows/pull-requests.yaml
vendored
|
|
@ -29,6 +29,7 @@ jobs:
|
|||
build:
|
||||
name: Build
|
||||
runs-on: [self-hosted]
|
||||
timeout-minutes: 30
|
||||
permissions:
|
||||
contents: read
|
||||
packages: write
|
||||
|
|
@ -150,6 +151,7 @@ jobs:
|
|||
name: "E2E Tests"
|
||||
runs-on: ${{ contains(github.event.pull_request.labels.*.name, 'debug') && 'self-hosted' || 'oracle-vm-24cpu-96gb-x86-64' }}
|
||||
#runs-on: [oracle-vm-32cpu-128gb-x86-64]
|
||||
timeout-minutes: 120
|
||||
permissions:
|
||||
contents: read
|
||||
packages: read
|
||||
|
|
|
|||
Loading…
Add table
Add a link
Reference in a new issue