[tests] Stabilize E2E kubernetes tests (#2262)

## What this PR does

Reduces flakiness in E2E tests by addressing the most common failure
modes.

### Kubernetes tenant tests (`run-kubernetes.sh`)

- Increase node Ready timeout from 2m to 5m — CI runners are shared and
resource-constrained
- Fail fast when nodes are not Ready — saves ~7 minutes per failed
attempt by not running LB/NFS tests that will also fail
- Delete stale Kubernetes resources at test start — retries provision
from scratch instead of patching stuck state
- Wait for port-forward to be ready before using it (fixes race
condition)
- Reduce version check polling interval from 5s to 1s

### All app tests (postgres, redis, kafka, clickhouse, mariadb, mongodb,
qdrant, foundationdb, openbao, vminstance, bucket)

- Add pre-cleanup of stale resources from previous failed retries so
each attempt starts clean

### Test runner (`cozytest.sh`)

- Clean up temp directory on test failure (was only cleaned on success,
leaking `/tmp` dirs)

### Release note

```release-note
NONE
```

<!-- This is an auto-generated comment: release notes by coderabbit.ai
-->
## Summary by CodeRabbit

* **Tests**
* Improved e2e robustness by deleting stale resources before creation
(ignore-if-missing, bounded time) and killing leftover port-forwards.
* Added an exit cleanup to consistently remove temporary artifacts and
teardown port-forwards on any exit.
* Added readiness polling for forwarded endpoints (curl with timeout);
made API/version retries more responsive.
* Extended node readiness wait (2m → 5m), dump debug info and fail fast
if unready.
  * Made load‑balancer reachability checking explicit and more reliable.
<!-- end of auto-generated comment: release notes by coderabbit.ai -->
This commit is contained in:
Kirill Ilin 2026-03-26 18:57:14 +05:00 committed by GitHub
commit dfe16ed9b4
No known key found for this signature in database
GPG key ID: B5690EEEBB952194
15 changed files with 58 additions and 22 deletions

View file

@ -29,6 +29,7 @@ jobs:
build:
name: Build
runs-on: [self-hosted]
timeout-minutes: 30
permissions:
contents: read
packages: write
@ -150,6 +151,7 @@ jobs:
name: "E2E Tests"
runs-on: ${{ contains(github.event.pull_request.labels.*.name, 'debug') && 'self-hosted' || 'oracle-vm-24cpu-96gb-x86-64' }}
#runs-on: [oracle-vm-32cpu-128gb-x86-64]
timeout-minutes: 120
permissions:
contents: read
packages: read