Consultants' 'Research' Isn't Science: Three Red Flags to Check

When Enron collapsed in 2001 — after Fortune had named it "America's Most Innovative Company" for six consecutive years — Nobel laureate Paul Krugman blamed neither corruption nor hubris. In his New York Times column "Death By Guru," he pointed at trendy management theories: consultants selling compelling narratives that persuaded CEOs to abandon fact-based analysis. The claim is worth stress-testing, because the deeper problem is structural. When a big consulting firm sells a story, it presents its assertions as "research" — yet it rarely, if ever, means the scientific method with appropriate controls for error and bias.

Here are the three techniques that deserve scrutiny.

What's wrong with the case study?

Consultants lean on case study interviews because organizations are inscrutable and hard to research. The pitch: rare access to insiders yields insight others cannot get. Sometimes that works. Often it doesn't.

Three limitations get overlooked:

• Survivorship bias. Managers answer to shareholders and customers, so they rarely discuss failures. If a risky strategy makes 1% of firms wildly successful and drives 99% out of business, we hear glowing accounts from the lucky 1% and miss the ones that flamed out.
• Thin samples. When researchers wrote a case study on a company the source author used to run, they interviewed only a handful of people — almost all from a single part of the business, and none besides the CEO involved in decisions. The issues in the case study weren't the ones management actually wrestled with.
• No documentation. Sociology and psychology researchers must share methods and data so others can interpret them. Consulting research rarely does.

Note what's missing here: the consultants aren't lying. The method itself can't carry the weight.

Why do executive surveys look scientific but aren't?

The second technique produces findings like "57% of companies are in the emerging stage of digital transformation" (Deloitte) and "49% of C-suite executives plan to diversify their supply chains" (Accenture). Numbers, graphs, pie charts — the visual grammar of a scientific paper.

The rigor is largely a mirage. The reports often omit how respondents were selected, how questions were phrased, how representative the sample was, and how answers were interpreted. Strip those details out and you have an opinion poll dressed as empiricism.

Worse, surveys measure what typical executives think — the current zeitgeist, which may or may not reflect market reality. As the source puts it bluntly: a great business strategy does not merely reflect what typical executives know, but exploits what they do not. Surveys can tell you what executives believe. They cannot tell you whether those executives are right or even well informed.

The mathematician G.H. Hardy said it a century ago: "It is never worth a first-class man's time to express a majority opinion. By definition, there are plenty of others to do that."

Who is actually speaking at industry conferences?

The third channel: the industry conference, where "industry experts" share a stage with the firm's in-house gurus. The promise is exposure to the latest research and trends. The reality, frequently, is that the executives giving case study interviews and answering surveys are the same people onstage and in the audience.

Consulting firms are selling their clients' ideas back to them. The conferences tout "out-of-the-box thinking" while building echo chambers for narratives the firm already pushes. Executives get wined and dined; a consensus is forged; nonconforming ideas are subtly discredited.

That doesn't make the ideas bad, and consultants often do outstanding work. But the record includes consulting firms behind Enron, Purdue Pharma's opioid promotion, and Allstate's efforts to lowball insurance claims — strategies that ended in scandal. The simple truth: consultants focus on making clients happy. Sometimes that means hard truths. Sometimes it means what the client wants to hear.

How do you spot pseudo-science in a deck?

In his 1974 Caltech commencement speech, Richard Feynman recounted watching a man at a new-age resort rub a woman's big toe and ask the instructor, "Is this the pituitary?" Feynman's retort: "You're a hell of a long way from the pituitary, man!" His point: official-looking charts and tables make something appear scientific without making it science.

Consultants boast of hundreds of case study interviews and databases covering thousands of firms. Without proper methods and controls, those numbers are meaningless — the statistical equivalent of reflexology.

There's a personnel problem too. Top-tier consultants arrive with blue-chip résumés from top schools, yet very few have operated a business. Many have been told they're smart all their lives, have never failed, and expect others to be impressed by their ideas rather than examine them.

What should buyers of research do?

The world is messy. The most important assets — relationships, trust — don't fit neatly into Excel spreadsheets. Ideas can only be validated forward, never backward, and countless logically sound ideas still fail in the marketplace.

So treat claims of "scientifically backed" or "research-based" strategy with a grain of salt. An idea is just an idea. The marketplace, not the pie chart, will tell you whether it's a good one — and anyone selling certainty before that test is asking you to skip it.