A Study is Not a Study

All scientific studies are not created equal.

Each type of scientific study has its uses and value, but when you make decisions on diet, exercise and other behaviors you are literally betting your life.

So it’s important to understand what different kinds of studies can tell us, and what they can’t.

The weakest evidence comes from observational studies. In a retrospective observational study, scientists compare outcomes in different populations or subgroups and compare characteristics of those populations.

They may note differences in outcomes, and then try to identify differences between the two populations that might explain the differing outcomes.

That’s an important and highly beneficial use of these kinds of studies: to generate hypotheses.

But those hypotheses need to be tested. Just because two factors are correlated doesn’t mean there is a causal relationship.

Sometimes the correlation is so strong that causation is highly likely. For cigarette smokers, for example, the incidence of lung cancer is 20-30 times higher than among nonsmokers. If the effect is large enough, an observational or epidemiological study can provide a good guide to decisions.

It would be foolish to think smoking is safe just because a controlled experiment in humans is impractical and unethical.

On the other end of the spectrum from observational studies is a randomized, double-blind and placebo-controlled trial.

  • A controlled study is one in which a group that is treated is compared with one that is not, the controls. If the outcomes of the treatment group are significantly better than the control group, it shows that the treatment has value.
  • Placebo-controlled means that some kind of intervention is done in each arm of the study. One group gets the pill that is hypothesized to be beneficial, while the other typically gets an identically appearing sugar pill. This is designed to overcome the placebo effect, in which people report feeling better because they think they’ve been treated. To be meaningful, the treatment arm needs to have results that are significantly better than placebo.
  • A randomized trial is one in which people are assigned by chance to either the treatment or the control.
  • A double-blind study is one in which neither the study subjects or the investigators know who is getting the treatment and who is a control. In addition to guarding against the placebo effect among subjects, this is designed to keep the investigators from imagining improvements in the treatment group.

This kind of study is often called a “gold standard” study. In the best of them, the treatment and control groups will be as identical as possible. You wouldn’t want one to have 60% tobacco smokers while the other has 30%, for example. Ideally, the only relevant difference between the groups will be that one got the treatment while the other didn’t.

It’s also important to have a large enough number (or n) to enable statistical analysis of the likelihood that the result was not due to chance. As you read studies you’ll note that a p value is given, such as p<.05. That means the likelihood that a difference between the groups is due to chance is less than 5%: we’re 95% certain that the outcomes difference is a real one.

The strongest studies also have “hard” or objective endpoints, which are the outcomes being measured. Death or a heart attach are hard endpoints, while subjective measures like “feelings of well-being” are soft endpoints.

One of the big problems in scientific reporting is when headlines say, “Study shows….”

If it’s a gold standard study (randomized, double-blind, placebo-controlled), that’s legitimate.

If it’s an observational or epidemiological study, the more accurate headline would be “Study suggests ….” Those studies can only generate hypotheses to be tested in more rigorous gold standard studies.

So why even do these observational studies? They enable identification of promising lines of inquiry at lower cost. They can provide a good starting point.

One final type of study that is relevant for our consideration is animal studies. Because of common metabolic pathways, these also can suggest potential applications in humans.

Like observational studies, animal studies can point to areas for further investigation, but they aren’t definitive. Their major advantage is that with the shorter lifespans of animals and our ability to enforce compliance with the intervention, we’re able to get those preliminary answers in a shorter timeframe. And we don’t need to worry about the placebo effect in animals.

But lots of findings in mice have not borne out in human studies.

So how does this relate to dietary studies, and what should we do about it?

  1. It’s almost impossible to do a study of free-living humans for any length of time that truly isolates one variable.
  2. The effects identified in observational studies are typically not large enough to justify confident proclamations. An odds ratio of 20 or 30 meant cigarette smokers were 20-30x more likely than nonsmokers to develop lung cancer. If the odds ratio for developing colon cancer because of nitrates in bacon is 1.1, it’s a lot less clear that bacon is really a problem (see #1).
  3. The effects of diet accumulate over a lifetime. Even smoking seems to have its carcinogenic effects over a period of years, not weeks or months.
  4. It’s extremely hard to have a blinded study of diet, at least with normal foods. People know whether they’re eating eggs or pancakes, bacon or okra.
  5. Eating in a way that is radically different from what our ancestors ate, at least over the last several thousand years, is unlikely to be a prescription for health and vitality.

My main point is that it’s hard to determine precisely the health effects of different eating patterns and habits. It’s even harder when one hypothesis, the diet-heart hypothesis, becomes established and leads to confirmation bias driving out dissent, making it difficult to do and publish research that doesn’t align with prevailing thought.

And yet, we each are our own n of 1, and we need to decide what we’re going to eat today. Based on my reading and personal study and experimentation over the last few years, I’m convinced that the diet-heart hypothesis is seriously off track.

As I continue this series, which you can follow on Facebook, Twitter  or LinkedIn (or by just bookmarking and checking in regularly), I’ll introduce you to authors, physicians and scientists who have influenced me toward that position, as well as to some of the stronger studies that have led me to adjust my personal behaviors.

Even if you reach a different conclusion, it’s good to have explored the arguments so you’re making dietary decisions consciously instead of by default.

I’m an n of 1…and so are you

When physicians and scientists are confronted with a case that conflicts with what they understand to be true, particularly in the areas of diet and exercise, one typical response is, “Well…that’s an n of 1.”

What they’re saying is essentially this: I can’t prove that thing didn’t happen. But if it did happen, it was in this one case. And to rely on it, we need to see it replicated in studies of a larger population.

So the fact that my LDL and Triglyceride levels dropped 20-30 points and my HDL (good) cholesterol went up 17 points after I switched from cereal to eggs as my breakfast staple doesn’t prove anything.

I’m an n of 1.

Fair enough. The problem, however, as Gary Taubes (more on him in future posts) and Nina Teicholz have pointed out, is that dietary studies are among the least reliable ones in the scientific literature.

They’re either short-term, which by definition means they can’t reliably measure long-term effects, or they tend to rely on self-reported data on food consumption.

How many servings of fruits and vegetables did you have last week? How confident are you in that figure? Do you think the recollection accuracy of dietary study participants is markedly more reliable?

For more than 40 years Americans have been told to eat more whole grains and avoid saturated fat, and the weight of government has been placed behind these directives. As Teicholz and Taubes have suggested, however, these recommendations were made with an air of certainty that was far beyond what the study data could justify.

In their defense, the dietary experts felt a need to give some kind of direction to those who wanted to eat healthy. But the result has been our whole society has been subject to dietary experimentation on a massive scale.

You’re in the same position as those experts of a generation ago, except you don’t have to make recommendations for society as a whole.

You just need to decide how you’re going to eat.

Our twin epidemics of obesity and diabetes suggest that confidence in our current dietary guidelines is misplaced.

You’re an n of 1, too. We each have to reach our best judgment of what eating habits will best promote our health and vitality.

In future posts I’ll highlight some studies and perspectives you might find helpful. All studies are not created equal. More on that next time.

See the whole series about my health journey. Follow along on FacebookTwitter and LinkedIn.

“You eat eggs for breakfast? Every day?”

Of course not.

Only on the days I eat breakfast.

“But what about your cholesterol levels? Isn’t that bad for you?”

Let me show you from my medical records.

I wasn’t getting my lipids checked all that regularly in my 40s, but from about age 46 to 53 my breakfast routine was essentially the same, usually having Corn Chex, or occasionally Cheerios or another gluten-free cereal.

So I have to believe that these results from July 2012 are representative of my lipid values during that period. And it’s interesting to compare them to the results from September 2018, after I had restored eggs, butter and cream to my diet:

After replacing cereal with eggs as my breakfast staple…

  • My LDL (bad) cholesterol went from 149 to 126
  • My HDL (good) cholesterol went from 47 to 64
  • My Triglycerides went from 87 to 57.

My blood pressure, which previously had been in the 130-140/90 range, is now consistently around 120/80.

Part of the explanation for better lab values is obviously that I’ve lost 40 pounds (at least) of fat while also gaining muscle.

But that’s kind of the point, right? I mean, that was the goal.

We’re told we should avoid animal fats because it will raise heart disease risk. We shouldn’t do the Atkins diet because it is’t heart healthy.

But isn’t being overweight (or borderline obese) a bigger risk?

As I look at my lab values, they’ve all moved in a positive direction.

I enjoy eggs. I don’t get hungry all day. And my blood tests look better than when I was eating “healthy whole grains.”

Seems like a win-win to me.

See the whole series about my health journey. Follow along on FacebookTwitter and LinkedIn.

4-Hour Body: Occam’s Protocol and the Minimum Effective Dose

In mid-2016 when Lisa suggested that I might consider adding weightlifting to my 30-minutes of daily cardio, I couldn’t see a way to make it work into my schedule.

After reading The 4-Hour Body, however, I had new inspiration to give it a try.

As Tim Ferriss explained the concept of a Minimum Effective Dose of weightlifting (as well as other interventions), it made sense to me.

To add muscle…do the least necessary to trigger local (specific muscles) and systemic (hormonal) growth mechanisms

Tim Ferris, The 4-Hour Body

The idea is that you need to stress your muscles for a relatively short period (he said about 80 seconds) to trigger an adaptation response.

More than that is not only wasteful, but may even be harmful.

Ferris also describes what he calls “Occam’s Protocol” which involves two alternating every other day between two daily weight-training workout that take about 20 minutes each.

I adapted it for my purposes, using equipment available at the YMCA:

  • Incline press (one set…increasing weight when I could do seven reps)
  • Pull down (same approach on reps and going up in weight)
  • 10 myostatic crunches (using a Bosu ball for full range of motion)
  • 10 Cat Vomits (gotta get the book for a description of that one!)
  • 25 minutes of cardio on the Precor elliptical machine

Later, instead of the incline press on a weight machine, I started doing a dumbbell chest fly (although probably not with very good form.) Still, it was my first real work with free weights.

I didn’t lift every day, because I knew days off for recovery were important. And I still wasn’t doing leg work, because I didn’t like the feel of the leg press machine and I had tight hamstrings. I figured the elliptical training was enough.

And I was still making it all fit in 30 minutes a day.

Still, by starting to add some muscle I was beginning body recomposition. The number on the scale wasn’t going down as quickly, but I was becoming fitter. And adding muscle meant my basal metabolic rate would increase.

See the whole series about my health journey. Follow along on FacebookTwitter and LinkedIn.

4-Hour Body: The Slow Carb Diet

Because of my interest in productivity sparked by David Allen and GTD, I also had encountered The 4-Hour Workweek by Tim Ferriss. And while I really liked my job and wasn’t looking to Escape 9-5, Live Anywhere and Join the New Rich, I found lots of his life hacks helpful.

What I really appreciated about his approach is his relentless desire to get the absolute best results from the minimum effort.

So given my newfound interest in a total body makeover (and Lisa’s frustration with the Trim Healthy Mama Plan), I checked out The 4-Hour Body.

This book is the result of Ferriss’ constant self-experimentation and measurement of results, to the point of eating two different ways on consecutive weekends, and weighing the resulting excrement. Or buying an ultrasound device that enabled him to make precise body fat measurements.

I figured I could learn from his experiments and recommendations.

So Lisa and I tried what Ferris calls The Slow-Carb Diet – Better Fat-Loss Through Simplicity. It’s all summed up in five simple rules.

  • Rule #1: Avoid “White” Carbohydrates – bread, rice, cereal, potatoes, pasta, etc. Cauliflower is the only exception.
  • Rule #2: Eat the same few meals over and over again. Mix and match from three groups that include proteins (eggs or meat), legumes (beans), and vegetables (spinach, broccoli, cauliflower, green beans.)
  • Rule #3: Don’t Drink Calories. No milk, soft drinks or fruit juice. Limit cream in coffee to two tablespoons.
  • Rule #4: Don’t Eat Fruit. Heads of dietitians everywhere are exploding at that one. As Tim says, though, my northern European ancestors didn’t get fruit in the winter, and yet here I am today.
  • Rule #5: Take One Day Off Per Week. On “cheat day” you can (and should) forget Rules 1-4, and eat whatever you want. Maybe even eat until you’re feeling a little sick.

We found the idea of a cheat day extremely helpful. Tim says it’s good for jolting your system, and preventing your body getting into caloric deprivation mode, which slows your metabolism.

But just from a psychological perspective it’s also great. As he says, when you go “on a diet” you will eventually binge. Why not schedule it in advance? Then you aren’t demoralized by failure.

Also, it makes it easier to comply during the week when you know you have cheat day coming. If you say, “I can never have ice cream again!” it seems onerous. Saying, “I can’t have ice cream now, but I can on Sunday!” is doable.

Tim also recommended getting at least 30g of protein within 30 minutes of waking in the morning. Lisa doesn’t like eggs, so she typically got hers from a protein shake.

For me, at least six days a week, I would scramble these together in a skillet for breakfast :

  • 3-4 eggs
  • 3 strips of chopped bacon or some diced ham
  • A handful of spinach
  • 1/2 cup of black beans

After this was cooked and on my plate, I typically topped it with salsa.

Lisa had a harder time with this diet because, unlike the Trim Healthy Mama Plan, there wasn’t a cookbook to help with dinner planning.

But while she only lost a pound per month with the other diet, she lost 10 pounds in three months on the Slow Carb Diet.

I continued to lose weight, about 10 pounds, during this time. But it was probably 15 pounds of fat loss, offset by muscle gain.

In my next post I’ll share what I learned about building muscle through this book , and how that got me started for the first time on a weight-training regimen.

See the whole series about my health journey. Follow along on FacebookTwitter and LinkedIn.