Why “evidence-based” is meaningless and we should stop using it

Whether policy or programs, “evidence-based” is the mantra. In a wide range of disciplines, many would go as far to say that evidence-based is fundamental.

It “helps nations bring about meaningful reforms and policy improvements that, in turn, can increase government accountability and facilitate performance-based budgeting and management,” as articulated by the World Bank. In short, effective governing should be based on evidence-based policy-making.

The same can also be said about public health policy; the most effective public health policies and programs are evidence-based. Right?

But what does “evidence-based” really mean?

For one, if you ask this question of 10 professionals from public health or public policy, chances are you won’t hear the same definition twice.

Evidence-based is a process term. It suggests there is something (i.e. some evidence) underpinning a policy or program. While this sounds rationale, the process by which we use evidence (in any form) doesn’t say anything about the quality, quantity, or consistency of the evidence itself.

As Jonathan Fielding, the former director of the Los Angeles County Health Department and now professor at UCLA, stated in a 2006 Health Affairs article,

“A person’s hunch, an anecdotal observation, the expert opinion of a group, a formally designed and executed scientific study, or a group of studies all can constitute evidence.”

So, one can describe or advocate for a particular policy or program with the same “evidence-based” argument when the actual quality of evidence that justifies it varies widely. Whether something is low-quality evidence, like opinion or testimonials, or high-quality, as in the case of large randomized clinical trials, they both receive the same stamp of approval.

cm-evidence-listed-medicines-05-01

Let’s look at how this practically plays out.

I’m a Member of Congress and I receive several letters from my constituents about how awful the new school lunch standards are. The constituents’ children don’t like the taste of the food and often throw it out. Despite higher quality evidence suggesting that the new school lunch standards increased the availability of healthier options, I’m convinced by my constituents’ stories. So, I push back against proposed 100% whole grain requirements, which ultimately get dropped from the 2016-17 federal omnibus appropriations bill.

In theory, this is still an evidence-based policy. I acted based on evidence (albeit not representative or higher-quality than other available evidence) provided by my constituents.

Here’s another, more common example. The Dietary Guidelines for Americans form the basis for nutritional advice by healthcare professionals, dietitians, educators and many others. They also underpin a wide range of nutrition policy in the US, like school meal standards. Yet, as the most recent process for the 2015 iteration highlighted, lower-quality observational studies can receive preferential treatment over higher-quality randomized controlled trials. The result is a recommendation based on hand-picked evidence, rather than the totality of evidence.

As one of the nation’s leading cardiologists, Dr. Steven Nissen of the Cleveland Clinic, points out that the new guidelines fail “to distinguish between recommendations based on expert consensus rather than high-quality RCTs [randomized controlled clinical trials].” In the case of dietary fat and cholesterol, this makes a huge difference. He argues, “The best available evidence does not clearly support the widely held belief that Americans should limit saturated fat and cholesterol in the diet.”

Here’s what Dr. Nissen is talking about:

  • A UCLA study of more than 136,000 patients nationwide found 75% of those hospitalized for a heart attack had cholesterol levels within “normal” ranges.
  • A 2015 systematic review concluded, “saturated fats are not associated with all cause mortality, CVD [cardiovascular disease], CHD [coronary heart disease], ischemic stroke, or type 2 diabetes.”
  • Another 2015 systematic review found that “in weight loss trials, higher-fat weight loss interventions led to significantly greater weight loss than low-fat interventions.”
  • And a 2012 review of evidence on dairy fat consumption and health risks found that “the majority of observational studies have failed to find an association between the intake of dairy products and increased risk of CVD, coronary heart disease, and stroke, regardless of milk fat levels.”

Yet, many forcefully contend that the Dietary Guidelines for Americans are, in fact, “evidence-based.”

Every policy or program has some rationale behind it. There’s always some “evidence,” even if it’s a collection of testimonials or anecdotes in support of it.

But don’t mistake the term evidence-based for unbiased or reflecting the highest quality of available scientific research.

At the end of the day, evidence-based speaks only of the process, and there are a lot of human choices that influence how this process plays out.

Advertisements

Leave a Reply

Fill in your details below or click an icon to log in:

WordPress.com Logo

You are commenting using your WordPress.com account. Log Out / Change )

Twitter picture

You are commenting using your Twitter account. Log Out / Change )

Facebook photo

You are commenting using your Facebook account. Log Out / Change )

Google+ photo

You are commenting using your Google+ account. Log Out / Change )

Connecting to %s