Diet and lifestyle changes are ideally made with the guidance of an experienced and trusted practitioner, but sometimes that’s impractical or impossible. Sometimes, you don’t have a practitioner like that near you, or you haven’t found one who suits you yet. Sometimes, you hear so many good things about a particular diet, exercise plan, or herbal remedy, you just have to see if it lives up to the hype. So you decide to give it a try for a while and see how it affects you—you set out to conduct an experiment on yourself.
It sounds simple enough, but there are a number of pitfalls with self-experimentation that can cause you to come up with confusing, incomplete, or unjustified conclusions. As a result, you might continue with a particular behavior that isn’t really helping, or you might dismiss as ineffective something that was actually beneficial—in both cases, defeating the purpose of your experiment.
How can this be prevented? First, we need to recognize the potential mistakes. Then we can find tools and strategies to help us avoid them.
self-experiment or self-justification?
There are some valid concerns about using self-experimentation as a technique for sorting out healthy and unhealthy diets and habits. Foremost among them is the prodigious human capacity for self-delusion and susceptibility to various forms of cognitive bias.
Andrew of Evolvify has written about what he calls “the self-justification diet”: “Bias makes it hard enough to train scientists to draw useful insight from experiments on others. Turning everyone into objective simultaneous experimenters and experimentees strikes me as an infinitely utopian endeavor.”
Chris Masterjohn comments in the same post: “A proper self-experiment, as opposed to a flimsily interpreted experience, would be to put oneself (preferably blinded with the help of someone else if possible) through several treatment and control trials in a randomized order, in which the “n” becomes the number of trials rather than the number of individuals. Then one can use statistics to determine if there is a treatment effect.”
Kurt Harris suggests that evidence from self-experimentation should indeed be the “ultimate” consideration—in the sense of the last thing considered, arguing that subjective experiences are too unreliable to be trusted and that scientific evidence should be the basis of all dietary and lifestyle changes. He goes on to say that “the scientific reasoning should be sound and there should be evidence. And then N=1 should be applied with caution, with emphasis on objective measures of health.”
If you followed any of the links in the last section, you saw the phrase n=1 come up a lot. In the current vernacular of the “paleosphere” (bloggers writing about various evolutionary-history-inspired diets, where this topic comes up often), n=1 is the shorthand used for self-experimentation. The term comes from the notation used in scientific studies (specifically, cross-sectional studies, or studies that take a single set of measurements from a number of participants) to indicate how many participants were involved: a study with 42 participants would be n=42. So, any self-experimentation would naturally be an n=1 experiment.
As you may have noticed, n=1 often comes with a pejorative undertone in these discussions. The criticism is that such studies can’t be used to generate reliable conclusions, because they don’t gather enough data to make generalizations. No reputable journal would publish a study that was only n=1 — “that’s not data, it’s an anecdote.” But this criticism is a red herring.
Ned Kock of Health Correlator wrote a nice article not too long ago, explaining why calling self-experimentation n=1 is inaccurate. In reality, he explains, n equals however many data points you give yourself; you are in fact doing a longitudinal study (a study that takes several measurements from participants over a period of time) with one participant. Just like with cross-sectional studies, greater n values are more useful. So if your “n=1” really is an n=1 — if you really only use a single experience (data point) as the basis for your conclusions, then yes, you’ve only reached the level of anecdote and you’re not justified in making claims. But if your “n=1” is really an n=10 or n=100, you’re that much more justified. (Refer back to Chris Masterjohn’s comment quoted above, where he says that “n” becomes the number of trials rather than the number of individuals.)
Why does this settle the criticism? Because it changes the terms of what we call n=1 and what we expect of our self-experimentation. We might still use the term n=1, but only in its colloquial sense; self-experimentation is the more exact term, and we can set down some parameters for what does and does not count as “real” self-experimentation.
subjectivity, objectivity, and practicality
An immediate reaction to first feelings (e.g. “but I feel good when I eat pizza!”) doesn’t constitute an experiment—you haven’t done the work. Delayed-onset and invisible effects haven’t been accounted for: are you feeling an endorphin response due to your undiagnosed gluten allergy? Is your immune system attacking your own tissues because they bear some structural similarity to food proteins that are passing undigested through your damaged intestinal walls? Will you have a migraine tonight, or be constipated tomorrow? And if you are, will you remember that the last three times you ate pizza, you had the same symptoms?
It would be nice to know for sure. It would be nice if we could have objective, clear-cut answers to every dietary and lifestyle question we can come up with. But objectivity like that isn’t cheap (and so, for many, it isn’t accessible), and in many cases it isn’t even possible.
There are some constants, and some concretes: trans fats are bad, green leafy vegetables are good. But more to the point, there are some areas where a large range of individual variation comes into play. Those are precisely the areas where self-experimentation is most helpful and indeed needful—and where, conversely, reliance on (sparse, contradictory, disputed) evidence from studies and one-size-fits-all prescriptions are least helpful.
How many grams of carbs per day makes a diet “low carb”? Can raw dairy be part of a “paleo” diet? Should nightshades be avoided entirely, or can they be eaten regularly? Is white rice a “safer starch” than sweet potatoes? There is no consensus among researchers or enthusiasts about these questions and more like them. Many have never been studied at all—either they were never considered, or they were disregarded as obvious due to the assumptions of conventional wisdom. And in the cases where they have been, the variation between individuals is high.
In herbal medicine, there are similar questions that can be most easily answered (for a given individual) by experimentation. Will valerian help me sleep or keep me up all night? Will the licorice in this tea blend exacerbate my high blood pressure? Is skullcap or blue vervain better for my particular breed of anxiety? An herbalist will be able to see the factors that make one or the other answer most probable, but the interaction between person and plant is complex and often surprising. If you’re on your own and want to get started right away, you’ll just have to try it and see—and that’s no problem! With a little work, you can make sure your self-experiments are successful.
perception, reflection, and connection
If we boil the many criticisms in the anti-n=1 posts and their attendant comments down to their fundamental ideas, we get the following core argument:
- self-experimentation is reliant on subjective experiences
- subjective experiences are susceptible to cognitive bias
- cognitive bias leads to unchecked and incorrect conclusions
- ∴ self-experimentation leads to incorrect conclusions
In logical analysis of an argument, you can either dispute the premises themselves or show that the conclusion doesn’t follow from the premises. In this case, the argument is of simple and direct form (if A then B, if B then C, if C then D; ∴ if A then D), so we must concentrate on the premises. Three premises means three points of vulnerability:
- reduce reliance on subjective experience
- reduce susceptibility to cognitive bias
- justify conclusions more thoroughly
Which is to say: we must work to develop skills of perception, for subjectivity shades into objectivity with sufficient perceptive depth and breadth; reflection, for cognitive bias can [only] be overcome by cognitive discipline; and connection, for discerning a distinct pattern gives us greater certainty in our conclusions.
In the next article, we’ll start at the beginning: with the development of skillful perception.
Other articles in this series: