Ci-Cd on ferkakta.dev

Zero-touch multi-tenant deploys: removing myself from the critical path

Mon, 02 Mar 2026 09:00:00 -0600

I had provisioned two tenants when I realized the deploy process didn’t scale to three. Each tenant on ramparts runs three services – api-server, web-client (the React frontend), tenant-auth – each with its own Docker image in ECR. Deploying a release meant running gh workflow run deploy-tenant.yml -f tenant_name=acme -f action=apply -f update_images=true, then doing it again for the next tenant. With 3 services resolving per run and N tenants, I was the bottleneck. Not Terraform, not GitHub Actions, not ECR. Me, remembering which tenants existed and typing their names correctly.

Expression injection in GitHub Actions repository_dispatch — and the one-line fix

Fri, 20 Feb 2026 10:00:00 -0600

I was hardening a cross-repo deploy pipeline built on repository_dispatch when I found a textbook expression injection sitting in plain sight. The trigger workflow accepted a client_payload JSON object from the caller and dropped its fields straight into a run: block.

How repository_dispatch works

When you call the repository_dispatch API, you send a JSON body with an event_type and an optional client_payload — an arbitrary JSON object the caller defines. Your workflow reads it via github.event.client_payload.*.

Self-healing race conditions: when your CI/CD fails on purpose

Fri, 20 Feb 2026 11:00:00 -0500

Three app repos build Docker images and push them to ECR. On merge, each fires a repository_dispatch to an infra repo’s orchestrator workflow. The orchestrator resolves ALL service images — not just the one that triggered it — and deploys every tenant via Terraform.

What happens when two repos merge at the same time?

The sequence

T=0: web-client and tenant-auth both merge to releases/0.0.2.
T=2m: tenant-auth build finishes first, fires dispatch.
T=2.5m: Orchestrator Run A starts. Tries to resolve all 3 service images.
T=2.5m: web-client image doesn’t exist yet — still building. Run A fails at image resolution.
T=4m: web-client build finishes, fires its own dispatch.
T=4m: Orchestrator Run B starts. Run A already finished (failed), so the concurrency group is free.
T=4m: Run B resolves all 3 images. Both new ones exist now. Deploy succeeds.

The end state is correct. Both changes deployed. One workflow run failed. Nobody had to do anything.

Cross-repo auto-deploy with GitHub Actions: the orchestrator pattern

Fri, 20 Feb 2026 10:00:00 -0500

Two repos merged within seconds of each other. The first orchestrator run failed — web-client’s ECR image didn’t exist yet because the build was still running. The GitHub Actions log showed a red X, an error annotation, and a Slack notification I didn’t need to read.

Four minutes later, the second run deployed both changes. No retry logic. No manual intervention. Nobody touched anything.

I’d spent my day building a cross-repo deploy pipeline for a multi-tenant platform — three app repos pushing Docker images to ECR, one infra repo deploying the new tenant service images to EKS. The race condition was the first real test. It failed exactly the way I wanted it to.

Your CI/CD dispatch token can rewrite your infrastructure code

Fri, 20 Feb 2026 09:00:00 -0600

I built a cross-repo auto-deploy pipeline this week. Three app repos push Docker images to ECR, then dispatch a deploy event to the infra repo’s orchestrator workflow via repository_dispatch. Standard pattern.

The gotcha: fine-grained PATs need contents:write to call the repository_dispatch API. Not actions:write — contents:write. The permission that also lets you push code, create branches, and delete files.

My service token that should only be able to say “hey, deploy this” can also rewrite the deployment workflow it’s triggering. That’s not least privilege. That’s a door that’s three sizes too wide.

Your terraform apply is silently rolling back your container images

Tue, 17 Feb 2026 09:00:00 -0600

Every “deploy to EKS with GitHub Actions” tutorial solves the same problem: build an image, push to ECR, deploy it. The tutorial ends at “your pod is running.” Nobody talks about day two.

The silent rollback

Day two: you have a running EKS cluster with three services per tenant. You need to change an IAM policy. You open a PR, touch one line of Terraform, run terraform apply.

Your IAM policy updates. Your container images also update — to whatever was hardcoded in variables.tf as the default. That default was correct three months ago. Your services just rolled back to a three-month-old image and nobody noticed because the deployment succeeded.