Reliability on ferkakta.dev

An orderly EKS and Kubeflow upgrade path

Fri, 27 Feb 2026 00:00:00 +0000

When EKS extended-support pricing is on the horizon, upgrade planning gets emotional fast.

The worst time to discover platform ambiguity is when finance and timelines are both tightening.

Our first impulse was to ask, “how quickly can we upgrade?”

The better question was, “what order of operations prevents us from compounding hidden drift during upgrade churn?”

Why one-shot upgrades fail in controller-heavy stacks

On paper, “upgrade EKS then bump Kubeflow” sounds linear.

Drift is an availability bug

Fri, 27 Feb 2026 00:00:00 +0000

I used to think of drift as a config hygiene issue.

Annoying, expensive, embarrassing — but fundamentally administrative.

Then I watched two control-plane components fall into CrashLoopBackOff inside a production incident and realized the framing was wrong.

Drift is not a paperwork problem. Drift is an availability bug.

The incident looked like random failure

We were already deep in one fire: a Kubeflow Pipelines frontend image that kept reverting to an old tag.

Kubeflow is a version matrix, not a version

Fri, 27 Feb 2026 00:00:00 +0000

“What version of Kubeflow are we on?”

That looks like a simple platform inventory question.

In practice, it was one of the most misleading questions in our incident.

We had already fixed one visible symptom — image reconciliation behavior that kept reverting a frontend component — when we started asking version questions to prevent recurrence.

The expected answer was one number.

The real answer was a matrix.

The false confidence moment

The dangerous moment was not when something failed. It was when everything looked green enough to stop looking.