Making a Kopf operator idempotent: three-layer existence checks and the redisReady race

Fri, 20 Feb 2026 12:00:00 -0500

Our tenant operator provisions databases, cache users, and credentials for each tenant in a multi-tenant SaaS platform. PostgreSQL roles on shared RDS, ElastiCache RBAC users, SSM parameters with generated passwords. It worked exactly once per tenant. The second time it ran, it regenerated every password and overwrote every SSM parameter. Running services holding the old credentials immediately lost their database and cache connections.

This was the blocker for auto-deploy.

Every deploy was a coordinated outage

The orchestrator runs terraform apply for each tenant on every deploy. Terraform reconciles the Tenant CRD, which fires Kopf’s on_tenant_create handler. The handler doesn’t distinguish between “new tenant” and “existing tenant whose CRD was re-applied.” It generates fresh passwords, creates new PostgreSQL roles (which fail because the role exists, or worse, succeed and orphan the old one), and overwrites SSM parameters with credentials that no running pod knows about.

Kopf on ferkakta.dev

Making a Kopf operator idempotent: three-layer existence checks and the redisReady race

Every deploy was a coordinated outage