Multi-Tenant on ferkakta.dev

Your employees are tenants and you should bill them like it

Mon, 16 Mar 2026 14:00:00 -0600

I built a Lambda that enriches every Bedrock invocation with cost data and routes it to per-tenant CloudWatch log groups. Model ID, input tokens, output tokens, estimated cost in USD, all written to /bedrock/tenants/{tenant} so each customer’s AI spend is visible in near-real-time.

Then a developer on the team needed Bedrock access for local development, and I had a problem I hadn’t anticipated.

The invisible burn

The developer’s use case was reasonable. He was building features against the Bedrock API and needed to iterate against real models, not mocks. I created an SSO permission set with bedrock:InvokeModel and handed him the profile name.

I assumed GovCloud was AWS with a different region code. It took two weeks to prove me wrong.

Wed, 11 Mar 2026 23:00:00 -0400

I needed a GovCloud account for a multi-tenant NIST compliance platform. I’d been running commercial AWS infrastructure for months — EKS, Terraform, tenant provisioning, the whole stack. GovCloud would be the same thing in a different region. That was the assumption. It lasted about four hours.

The account that doesn’t exist yet

My management account couldn’t call CreateGovCloudAccount. The API returned ConstraintViolationException with a message about not being “enabled for access to GovCloud” and no guidance on what that meant. I filed a support case. AWS enabled the permission two days later, and as a side effect created a standalone GovCloud account that had no relationship to my Organizations structure — an orphan floating in the partition with disconnected root credentials. I still had to find it and deal with it.

From eight manual steps to one command

Tue, 03 Mar 2026 09:00:00 -0600

I provisioned two tenants by hand before I decided that nobody should ever provision a tenant by hand.

The provisioning flow for our multi-tenant SaaS platform was 8 steps across 4 tools — a Python CLI, a shell script with 5 flags per invocation, a GitHub Actions workflow, and two Kubernetes job manifests requiring injected DB connection strings. Each step had different inputs, different env files, and subtly different flag names for the same concept. The two populate runs used --appname apiserver and --appname tenant_auth_service — note the underscore in one and not the other. That naming inconsistency is a guaranteed typo on a Friday afternoon. Each flag is a chance to silently write 24 SSM parameters to the wrong path.

Your onboarding flow is your architecture's report card

Tue, 03 Mar 2026 00:00:00 +0000

I ran a colleague’s manual tenant onboarding flow for a multi-tenant SaaS platform. Five steps, two attempts, and a list of errors that mapped precisely to every automation gap in the system. The onboarding flow wasn’t broken. It was a diagnostic.

The five steps

The flow to bring a new tenant from nothing to working:

Run a Python registration script that calls the auth-handler API, creates an org in the identity provider, and sends a confirmation email to the devops team.
Read the devops email. Manually extract two values: a tenant hash and an org code.
Run populate scripts that seed 38 SSM parameters — 24 for apiserver, 14 for tenant-auth-service.
Trigger a GitHub Actions workflow. Terraform creates the namespace, deployments, ExternalSecrets, DNS records, HTTPS.
Manually apply Kubernetes jobs for ETL seed data and first-user creation.

Step 4 is automated. Steps 1, 2, 3, and 5 are manual. The manual steps are where the architecture’s seams show.

Zero-touch multi-tenant deploys: removing myself from the critical path

Mon, 02 Mar 2026 09:00:00 -0600

I had provisioned two tenants when I realized the deploy process didn’t scale to three. Each tenant on ramparts runs three services – api-server, web-client (the React frontend), tenant-auth – each with its own Docker image in ECR. Deploying a release meant running gh workflow run deploy-tenant.yml -f tenant_name=acme -f action=apply -f update_images=true, then doing it again for the next tenant. With 3 services resolving per run and N tenants, I was the bottleneck. Not Terraform, not GitHub Actions, not ECR. Me, remembering which tenants existed and typing their names correctly.

Per-Tenant CloudWatch Log Isolation on EKS, or: Why I Stopped Using aws-for-fluent-bit

Mon, 02 Mar 2026 00:00:00 +0000

The starting assumption

I’m building ramparts, a multi-tenant compliance platform running on EKS. Each tenant gets a Kubernetes namespace – tenant-acme, tenant-globex, whatever – and the compliance controls require that their application logs land in isolated storage with 365-day retention. CMMC maps this to AU-2 (audit events), AU-3 (audit content), AU-11 (retention), and AC-4 (information flow isolation). A tenant cannot read another tenant’s container output.

The obvious first move was aws-for-fluent-bit, AWS’s own Helm chart and container image for shipping logs to CloudWatch. AWS service, AWS chart, AWS logging destination. The blessed path.

Making a Kopf operator idempotent: three-layer existence checks and the redisReady race

Fri, 20 Feb 2026 12:00:00 -0500

Our tenant operator provisions databases, cache users, and credentials for each tenant in a multi-tenant SaaS platform. PostgreSQL roles on shared RDS, ElastiCache RBAC users, SSM parameters with generated passwords. It worked exactly once per tenant. The second time it ran, it regenerated every password and overwrote every SSM parameter. Running services holding the old credentials immediately lost their database and cache connections.

This was the blocker for auto-deploy.

Every deploy was a coordinated outage

The orchestrator runs terraform apply for each tenant on every deploy. Terraform reconciles the Tenant CRD, which fires Kopf’s on_tenant_create handler. The handler doesn’t distinguish between “new tenant” and “existing tenant whose CRD was re-applied.” It generates fresh passwords, creates new PostgreSQL roles (which fail because the role exists, or worse, succeed and orphan the old one), and overwrites SSM parameters with credentials that no running pod knows about.

Cross-repo auto-deploy with GitHub Actions: the orchestrator pattern

Fri, 20 Feb 2026 10:00:00 -0500

Two repos merged within seconds of each other. The first orchestrator run failed — web-client’s ECR image didn’t exist yet because the build was still running. The GitHub Actions log showed a red X, an error annotation, and a Slack notification I didn’t need to read.

Four minutes later, the second run deployed both changes. No retry logic. No manual intervention. Nobody touched anything.

I’d spent my day building a cross-repo deploy pipeline for a multi-tenant platform — three app repos pushing Docker images to ECR, one infra repo deploying the new tenant service images to EKS. The race condition was the first real test. It failed exactly the way I wanted it to.

90 AWS resources in 5 minutes — automating multi-tenant SaaS tenant lifecycle

Tue, 10 Feb 2026 09:00:00 -0600

I recorded our entire tenant lifecycle — create, test, destroy — with no edits. Here’s what 5 minutes of infrastructure automation looks like when there are no tickets, no handoffs, and no “can someone set up the database.”

What happens on `tenant create`

One GitHub Actions workflow backed by Terraform + a Kubernetes operator:

Validates the tenant name, resolves container images from the latest release branch
Provisions ACM wildcard cert + Route53 DNS records
Creates the Tenant CRD → operator provisions PostgreSQL databases on shared RDS, seeds credentials to SSM
Terraform deploys ExternalSecrets, Deployments, Ingress — 3 services per tenant
SSM parameters auto-seeded: Redis credentials, auth URLs, signing keys — ~40 config values per tenant
Zero static credentials anywhere — IRSA for everything, secrets injected at runtime from SSM via External Secrets Operator

About 5 minutes from nothing to 90 AWS resources and running pods.