Bedrock on ferkakta.dev

The Allow SCP that worked until it didn't

Tue, 24 Mar 2026 21:00:00 -0600

I run a multi-tenant SaaS platform on AWS with Control Tower managing the organization. Control Tower deploys a region deny guardrail — an SCP that blocks API calls outside your home region. The mechanism is a NotAction deny: it lists services that are allowed to operate globally (IAM, CloudFront, Route 53, a few dozen others), and denies everything else when aws:RequestedRegion doesn’t match your approved list.

This guardrail is one of the first things you hit when you try to do anything interesting. And the documentation says you can’t override a deny with an allow.

The $233 Day, Part 2: The Inference Iceberg

Fri, 20 Mar 2026 17:00:00 -0500

I posted the part 1 findings to the team thread — model switch, cache invalidation, 20× call volume, $173 training run. Case closed. The numbers were clean, the explanation was satisfying, and the model got reverted within the hour.

Except $173 was wrong. Not wrong in the analysis — the training run did cost that much. Wrong in scope. I’d found the visible part of the spend and stopped looking.

The $173 Training Run

Fri, 20 Mar 2026 15:00:00 -0500

The Slack message landed at 3pm on a Wednesday: “model training successful, previously 20min, now 1h30m.” I had finished an EKS 1.32-to-1.33 upgrade on the ramparts cluster that morning. My upgrade, my timeline, my problem.

The first theory wrote itself. New cluster version, fresh nodes, cold image caches. I’d fixed a broken cluster autoscaler earlier that day — the old autoscaler deployment was pinned to a node selector that no longer matched after the upgrade, so pods were stacking up in Pending until I caught it. First-run penalties after a major version bump are real. Everyone on the call nodded. I almost typed up that explanation and moved on.

Your employees are tenants and you should bill them like it

Mon, 16 Mar 2026 14:00:00 -0600

I built a Lambda that enriches every Bedrock invocation with cost data and routes it to per-tenant CloudWatch log groups. Model ID, input tokens, output tokens, estimated cost in USD, all written to /bedrock/tenants/{tenant} so each customer’s AI spend is visible in near-real-time.

Then a developer on the team needed Bedrock access for local development, and I had a problem I hadn’t anticipated.

The invisible burn

The developer’s use case was reasonable. He was building features against the Bedrock API and needed to iterate against real models, not mocks. I created an SSO permission set with bedrock:InvokeModel and handed him the profile name.