Incident Review

Tuesday. 2:02. incident review.

I have six slides for a twenty-three minute outage. Not so bad.

Title slide says BLAMELESS in small gray letters. Visible but not making a big thing of it.

Nemesis is across the table. He was on call. I was... involved.

Slide two.

1:06 checkout update set to 5%
1:06 error rate increases
1:09 alert fires
1:17 rollback begins
1:29 recovery complete

"The update was intended for five percent of traffic," I say.

Nemesis raises one finger. Not high. Just enough.

"Doesn't that client take a decimal?"

I know what he means.

"It takes both."

A lie.

"I mean the dashboard takes a number and the client converts it."

Nemesis looks at the timeline.

"So five became five hundred percent."

"It caps at one."

i am defending the cap.

"So one hundred percent," he says.

"Right. Safely capped." I wince.

manager rubs his eyes and writes something down.

Could be anything.

I move to slide three before the word safely can be discussed further.

Screenshot of the rollout dashboard. I added a red box around the 5 this morning. Useful visual aid against myself.

"Staging used point-zero-five," Nemesis says.

"Yes. The production value was entered in the displayed unit."

Displayed unit. Good. Passive enough to keep going.

"You entered five," he says.

"In a field showing a percent sign."

I zoom in.

There it is beside the input. Small gray %. The preview underneath says 5% of users.

Wait.

"The preview is wrong."

Nemesis leans toward the screen.

I click into the screenshot. Zoom? Enhance??

"The dashboard labels this as five percent, then sends the raw five. The client clamps it to one."

Our manager stops writing.

"So the interface confirms one unit and the API consumes another?"

"Yes."

This yes is much better than the first one.

Nemesis looks at the screenshot for another second.

"Right," he says. "Then we should validate it on both sides."

He says right very quickly.

Corrective actions.

Nemesis proposes a typed rollout field with a hard range check. Goes to me.

I propose alerting by rollout cohort so a small bad release cannot hide inside the global average. Goes to him.

We have exchanged chores. Blamelessness restored.

Meeting ends at 2:31.

We both have action items. No blame needed.

Such a small incident.