I debugged a Lambda timeout for 6 hours. The fix was 4 CLI commands.

Wed, 11 Mar 2026 16:00:00 -0400

The ticket said the Lambda tracer was timing out. The Slack thread said ConnectTimeoutError to an internal tracing endpoint. Four Lambda functions had been moved into a VPC the day before so they could reach tracer.internal.ferkakta.net — an internal ALB at 10.x.x.x, only reachable from inside the VPC. The migration was verified, the API returned success, the ticket should not have existed.

The people who built this system had moved on to other projects. The people using it were in a different timezone. There was no architecture doc, no runbook, no one to pair with. I had CloudWatch, a kubectl context, and AWS credentials.

IAM eventual consistency is 4 seconds — if your policy still doesn't work, you have a bug

Thu, 26 Feb 2026 09:00:00 -0600

I changed an IAM inline policy on a role — added an sts:AssumeRole statement so a pod could assume a cross-account SES role. Ran terraform apply. Checked the policy with get-role-policy. The old policy came back. No new statement.

I said “propagation delay” and moved on to other work.

Twenty minutes later I checked again. Same old policy. That’s not propagation.

What eventual consistency actually means

AWS IAM uses a distributed computing model. Changes to policies, roles, and credentials take time to replicate across endpoints. AWS documents this explicitly and recommends not including IAM changes in critical code paths.

Debugging on ferkakta.dev

I debugged a Lambda timeout for 6 hours. The fix was 4 CLI commands.

IAM eventual consistency is 4 seconds — if your policy still doesn't work, you have a bug

What eventual consistency actually means