Why Smart Brands Look Beyond the Dashboard | Upside #92

Strategy

25 Jun

Amazon Data Isn't Lying, But It's Incomplete

During WWII, the Ministry of Defence wanted to armour the parts of returning bombers with the most bullet holes. Statistician Abraham Wald said the opposite: armour where the holes aren't, because those planes never made it back. This episode uses that story to explain why your Amazon dashboard can be technically accurate and still lead you to the wrong decision.

In this episode: the team breaks down why the real edge isn't better data, since everyone has access to the same numbers, it's the judgment applied on top of it.

Key takeaways

The data isn't wrong, it's incomplete: a dashboard shows a narrow slice of what's happening and it's on the seller to ask what context is missing.
A real example: a sudden sales spike looked like strong ad performance, until checking the weather showed it coincided with two unusually warm, sunny days, confirmed against the previous year's data.
Regional weather matters more in larger, geographically split markets like the US and parts of Europe than in smaller ones, and it's the customer's weather that matters, not the seller's.
Competitor behaviour, stockouts, promotions, lightning deals, drives more of the swing in sales than most dashboards capture. Tracking competitor rank and promo cycles builds a working "psychological profile" that explains moves the numbers alone can't.
ROAS on its own can mislead: a strong ROAS paired with a low top-of-search percentage (say, 20%) can be quietly hiding several times the available growth in that keyword.
Even a category that's currently "flying" can call for pulling investment rather than doubling down, if historical seasonal data shows a decline is coming.
The actual competitive edge isn't the data, it's trained human judgment applied consistently on top of it, since every seller is looking at broadly the same numbers.

Timestamps

00:00 — The WWII survivorship bias story
00:52 — Why the data isn't wrong, it's incomplete
01:56 — The weather-driven sales spike
03:17 — Why regional weather matters more in bigger markets
04:50 — Competitor behaviour as a hidden data point
07:38 — Why ROAS alone can hide five times the available growth
08:53 — Pulling back from a category that's flying right now
11:33 — Why the real edge is judgment, not the data itself

The detail

The survivorship bias problem
Abraham Wald's WWII insight was that the planes worth studying were the ones that never came back, not the ones returning with visible damage. Applied to Amazon, the dashboard shows what happened to the traffic and campaigns that are already being tracked, but says nothing about the opportunities, or risks, sitting outside that view. The habit worth building is asking, on a regular basis, what isn't this data showing me, rather than only optimising what's already visible.

The weather-driven sales spike
A sudden two-day sales spike looked like a sign to pour more budget into what was working, until checking that period against the weather showed both days were unusually warm and sunny, and cross-referencing the previous year's data showed the same pattern during the sunniest April on record. Treating that spike as pure ad performance, rather than partly weather, could have meant pushing spend into campaigns right before a weather-driven drop wiped out the return.

Competitor behaviour as a hidden data point
Sales dashboards typically show a seller's own numbers, not what a competitor is doing that's quietly explaining the swing, a competitor low on inventory, running a lightning deal, or absorbing a stockout will all move sales in ways that look unexplained without that context. Tracking competitor rank and promotional cycles over time builds a working sense of how each one is likely to behave in a given season, turning otherwise confusing swings into an explainable pattern.

Why ROAS alone can hide real growth
A campaign with a strong ROAS of four, five or six can still be leaving significant growth on the table if its top-of-search percentage is low, say 20%, since that means four to five times the available impression share in that position is going untouched. Judging a campaign on ROAS alone, without checking top-of-search share, can make an under-performing opportunity look like a finished success.

When "it's flying right now" is the wrong signal
A product currently outperforming everything else in the account can still be the wrong one to invest further in, if historical seasonal data shows that category reliably drops off in the coming months. The harder, more useful move is often replacing that revenue with other products ahead of the decline, rather than chasing the current spike, even though pulling back from something that's visibly working is a genuinely difficult call to make.