ferkakta.dev

Three holes in the partition wall

I assumed GovCloud was AWS with a different region code. I wrote a whole post about how wrong that was. The partition wall between commercial AWS and GovCloud is real — no shared IAM, no cross-partition role assumption, no federated identity, no common STS endpoints. An arn:aws: principal cannot see an arn:aws-us-gov: resource. They are separate universes connected by a billing relationship and nothing else.

Except that’s not quite true either. There are three holes in the wall, and I found them one at a time over the course of a month.

Hole one: billing

When you create a GovCloud account through Organizations, AWS returns two account IDs. One is the GovCloud workload account. The other is a commercial “linked” account that exists solely for billing. There’s also a third — an AWS-managed account that holds encrypted secrets during the root key retrieval ceremony. Three account IDs for one “account.”

The linked commercial account has no workloads. It can’t run anything. It exists because AWS billing infrastructure lives in the commercial partition, and GovCloud accounts need to pay for things. The billing relationship crosses the wall because billing predates GovCloud — the commercial business existed first, and GovCloud was built inside it.

I spent an evening confusing these account IDs with each other. The linked account shows up in your commercial Organizations console as a member account with a name you didn’t pick and no way to sign in. It looks like a mistake. It’s the billing conduit.

Hole two: root credential delivery

GovCloud root users have no password. Not “shouldn’t use” — there is no password and no password recovery. Root access is programmatic-only via access keys. To get those keys, you file a support case from the commercial linked account, provide a KMS key for encryption, and wait for AWS to deposit the credentials in Secrets Manager.

The delivery is one-shot. get-secret-value succeeds once. I had to have a password vault open before running the command, because there was no second chance.

This took five support cases over two weeks. The first case went to the wrong account. The second used a KMS key in the wrong partition. The third went to the right account with the right key. Five rounds of “close this case and open a new one with the correct details,” each one a day of calendar time.

The root credential ceremony crosses the wall because identity is a customer-level concern, not a workload-level concern. AWS needs to prove that the person requesting GovCloud root credentials is the same person who owns the commercial account that pays for it. The partition isolates workloads, but someone has to own them, and that ownership lives in commercial.

Hole three: support

I discovered this one by accident. I’d submitted an SES production access request for GovCloud. AWS responded with “tell us more” on a support case. The case ID started with the linked commercial account number — 868550873125, not the GovCloud account 575826221478. The case lived in commercial even though the SES service lives in GovCloud.

I didn’t have console access to the linked commercial account. No SSO profile, no IAM user, no way to log in and reply through the web interface. But I tried the CLI anyway:

AWS_PROFILE=nistagent-govcloud AWS_REGION=us-gov-east-1 \
  aws support add-communication-to-case \
  --case-id "case-868550873125-muen-2026-..." \
  --communication-body "..." \
  --region us-gov-east-1

{"result": true}

The support API from GovCloud credentials can read and reply to cases on the linked commercial account. AWS Support is itself a partition boundary broker. I replied to a commercial-partition support case using GovCloud-partition credentials, and AWS processed it correctly. The response showed up in the case, the SES team read it, and they approved production access the same day.

The pattern

The three holes share a property. They’re all about the customer relationship, not the workload.

Billing exists because someone has to pay. Root credentials exist because someone has to own. Support exists because someone has to ask for help. None of these are workload operations. They’re customer operations — and the customer predates the partition. AWS built GovCloud inside an existing commercial business, and the commercial business has rights and needs that can’t be contained within the partition.

The partition isolates compute, storage, networking, identity, and every service that touches data. It does not — and cannot — isolate the human who pays for it all. Billing, support, and root identity are the three places where the customer’s existence is more fundamental than the partition’s existence.

Support also has to work this way for a simpler reason: there is no separate AWS GovCloud support workforce. AWS doesn’t maintain a parallel FedRAMP-cleared support organization for every GovCloud service — SES, EKS, IAM, billing, account management, all of it. The same support engineers handle both partitions. The API crossing the boundary is a reflection of the fact that the humans on the other side already cross it. The partition wall is a software construct. The people who maintain what’s on both sides of it were never partitioned.

I think this is deliberate and well-designed. GovCloud is FedRAMP High. The threat model assumes compromised workstations, hostile networks, nation-state adversaries. Every convenience that commercial AWS provides — ambient region configuration, persistent credentials, federated identity — GovCloud deliberately removes because those conveniences are attack surface. But the three holes remain because the alternative is a customer who can’t pay, can’t prove they own their account, and can’t ask for help. Those are worse than the holes.

Every other service I’ve tried — Route53 for public DNS, CloudTrail for organization-level trails, S3 Vectors, Lambda code signing before September 2025 — stops at the wall. The partition is real. It’s just not the whole story.

#aws #govcloud #platformengineering