Seven Questions to Ask Before You Believe a Sleep Study
In a 2008 study, people reported 6.8 hours of sleep that wrist sensors measured at 6.0. Seven checks for reading a sleep study, each tied to a real paper.
In this article11 sections
Sleepers in Chicago over-reported their nights by 48 minutes: in a 2008 study of 669 adults, people said 6.8 hours and wrist sensors measured 6.0, and that gap is the first of seven things a reader can check before trusting a sleep headline. A headline holds up to the extent that the reader can answer all seven: how sleep was measured, who and how many were measured, whether something else could explain the link, whether the result transfers from the species or setting tested, whether the outcome measured is the outcome claimed, how large the effect is, and whether anyone independent has repeated it. Most headlines fail at least one, usually visibly in the abstract.
The seven questions are ordinary critical-appraisal habits sorted into an order that suits sleep research, and each is tied to a published study you can look up. The order and grouping are this article’s choice, not an established standard, and nobody has tested whether applying them improves anyone’s judgment.
1. Was sleep measured, or remembered?
Most large sleep studies ask people how long they sleep. A 2008 paper in Epidemiology by Diane Lauderdale, Kristen Knutson, Lijing Yan, Kiang Liu and Paul Rathouz compared the two methods in 669 adults from the Chicago site of the CARDIA cohort, who wore wrist actigraphs for three days and also answered questions about usual sleep. Measured sleep averaged 6.0 hours. Reported sleep averaged 6.8 hours, and the correlation between the two was 0.47. People who actually slept 5 hours over-reported by about 1.2 hours; people who slept 7 hours over-reported by about 0.4.
Self-reported sleep duration is a moderately accurate and systematically optimistic estimate of measured sleep duration, and the error is largest in short sleepers. This matters for every study that sorts people into “under six hours” and “over eight hours” on the strength of a questionnaire: the short group contains people who sleep less than they say, and the groups are blurred in ways nobody can fully reconstruct. Wearables are not a clean fix. The post on how actigraphy turns wrist movement into a sleep label lays out why total sleep time is the one number those devices estimate well and why their wake estimates run generous. The problem is older than sleep science: Feynman described maze rats that used a cue the experimenter had overlooked, and that story is a compact picture of what remembered sleep does to a study.
2. Who was measured, and how many?
Two separate problems hide here. One is who is in the sample. The Lauderdale data come from a single city’s cohort of middle-aged adults. A 2015 study in Current Biology by Gandhi Yetish and colleagues recorded sleep in the Hadza of Tanzania, the San of Namibia and the Tsimane of Bolivia, and found average sleep of between 5.7 and 7.1 hours. That is a reminder that “normal sleep” in the literature is mostly normal sleep in industrialized, lab-adjacent populations.
The other problem is how many. In 2018, Ehsan Shokri-Kojori and colleagues at the US National Institutes of Health reported in PNAS that after about 31 hours awake, 20 healthy adults aged 22 to 72 showed roughly a 5 percent rise in beta-amyloid, measured by PET imaging, in regions including the right hippocampus and thalamus. PET scanning is costly, so 20 is a reasonable lab sample. It is also a sample in which a handful of unusual people can move the average, and spread across five decades of age it leaves little room to ask whether a 22-year-old and a 72-year-old respond alike. For that age question, the post on how sleep stages shift by decade is a useful companion.
3. Could the arrow point the other way, or is something else driving both?
In 2010, Francesco Cappuccio and colleagues published a meta-analysis in Sleep covering 16 prospective studies, 27 cohort samples, 1,382,999 participants and 112,566 deaths. Short sleep was associated with higher mortality (relative risk 1.12, 95% confidence interval 1.06 to 1.18). Long sleep was associated with a larger increase (relative risk 1.30, 1.22 to 1.38).
Two features of that result get lost in coverage. First, the data are observational and most of the sleep measures were self-reported, which brings back question 1. Second, the long-sleep signal is the stronger of the two, and nobody proposes that sleeping nine hours causes death. The usual reading is that illness, depression and other conditions lengthen time in bed while also raising mortality, which means the arrow can run from health to sleep as easily as the reverse. A study that finds a U-shaped curve has already shown you that sleep duration is partly a marker for something else. The short-sleep half carries the same uncertainty.
A pooled risk ratio from observational cohorts describes who died, not what sleep did. When a finding appears at both ends of a range, at least one end is probably not caused by sleep.
4. Does it transfer from the species and setting tested?
Lulu Xie, Maiken Nedergaard and colleagues reported in Science in 2013 that in mice, natural sleep and anesthesia were associated with a 60 percent increase in the interstitial space of the brain and faster clearance of beta-amyloid. The finding was widely summarized as “sleep cleans your brain.” A 2024 paper in Nature Neuroscience by Miao and colleagues at Imperial College London followed fluorescent tracers through mouse brains and reported reduced clearance in sleeping and anesthetized mice compared with awake animals. The two groups used different methods and reached opposite conclusions in the same species, and the dispute over the right method is still running.
That does not settle what happens in a human brain. A mouse is nocturnal and small, the measurements involve invasive imaging or tracer injections, and each step from rodent tracer data to a person’s routine has to be argued rather than assumed. When a headline says “your brain,” check whether the paper says “mice.”
5. Is the outcome measured the outcome claimed?
The Shokri-Kojori result shows this well. What was measured was a 5 percent change in a PET tracer signal after a night without sleep. What commonly reached readers was a statement about Alzheimer’s risk. The paper’s own framing links sleep loss to a risk factor for Alzheimer’s disease. A tracer reading after one night of total deprivation in healthy adults says nothing direct about whether years of ordinary short sleep lead to dementia, and the study did not follow anyone long enough to see disease develop.
The pattern recurs in sleep work: a proxy (a brain scan reading, a wearable’s “readiness” score, a questionnaire total) stands in for a clinical outcome (dementia, heart disease, depression). Ask for the sentence in the methods section that names what was actually recorded, then compare it with the sentence in the press release.
6. How large is the effect, not just whether it exists?
Statistical significance reports whether an effect is distinguishable from zero. It says little about whether the effect should change anyone’s habits. The risk ratio of 1.12 for short sleep in the Cappuccio meta-analysis is a 12 percent relative increase in risk over follow-up periods of 4 to 25 years. That can matter for public health across a population and still be a weak guide for any one person’s Tuesday.
A useful comparison comes from a neighboring field. Amy Orben and Andrew Przybylski, writing in Nature Human Behaviour in 2019, analyzed three datasets with 355,358 participants in total and found that digital technology use explained at most 0.4 percent of the variation in adolescent well-being. The association was real and negative, and small. With big samples, nearly any association is “significant.”
7. Who paid for it, and has anyone independent repeated it?
Funding is not proof of anything, but it predicts direction. A Cochrane review led by Andreas Lundh (2017) pooled studies comparing industry-sponsored drug and device research with other sponsorship and found that industry-sponsored studies more often reported favorable efficacy results (risk ratio 1.27, 95% confidence interval 1.17 to 1.37) and more favorable conclusions (1.34, 1.19 to 1.51). Within sleep medicine, Daniel Kripke wrote in the Journal of Clinical Sleep Medicine in 2007 that almost all trials of sleep-disorder drugs are industry-sponsored, citing a systematic review in which all but 5 of 56 randomized trials of insomnia drugs that reported funding were apparently industry-funded.
The commitment-device research this site cites tested other products in other populations. No published study has tested DontSnooze, so read the app’s claims as argument, not result: DontSnooze (iOS only).
Repetition is the second half of the question. Carrie Bailes, Mary Caldwell, Erin Wamsley and Matthew Tucker published a failure to replicate in PLoS ONE in 2020: sleep did not protect memories from interference better than waking did in their data. A meta-analysis by Nicola Newbury, Rebecca Crowley, Kathleen Rastle and Jelena Tamminen in Psychological Bulletin (2021) estimated that after adjusting for likely missing studies, the effect of sleep deprivation on memory was nearer g = 0.46 when deprivation came before learning and g = 0.17 when it came after, and the authors called for well-powered, preregistered replications to pin the sizes down.
Public books are not exempt. In 2019 the writer Alexey Guzey published a long critique of Matthew Walker’s Why We Sleep, arguing among other points that the World Health Organization never declared a sleep-loss epidemic, as the book says. Walker reportedly conceded that the WHO attribution was misremembered and traced to a 2014 statement from the US Centers for Disease Control and Prevention. A claim that has drawn a published, checkable critique and a response is in better shape than one nobody has examined, and with any popular book the cited sources, not the prose, are what to check.
Headline to question: a worked table
These headlines are paraphrases written for this article, not quotations.
| Headline (paraphrased) | Question it breaks | Why |
|---|---|---|
| “People who sleep under six hours die earlier” | 1 and 3 | Sleep came from questionnaires; illness and other conditions can sit behind both sleep and death |
| “Long sleepers face a 30% higher risk of death” | 3 | The arrow plausibly runs from poor health to long time in bed |
| “One bad night raises your Alzheimer’s risk” | 5 and 2 | A 5 percent PET signal in 20 adults, not dementia incidence |
| “Sleep washes toxins from your brain” | 4 | Mouse tracer data, with a 2024 mouse paper pointing the other way |
| “Screens damage teen well-being” | 6 | A clear association can still explain under half a percent of the variation |
| “New drug restores natural sleep, study finds” | 7 | Sponsor-run, placebo-controlled, no independent repeat |
| “How hunter-gatherers sleep proves we need nine hours” | 2 | Measured sleep in three such societies ran 5.7 to 7.1 hours |
What counts as a quick test for a trustworthy sleep headline?
A sleep headline passes a quick test when its source article reports measured rather than remembered sleep, a sample large and varied enough for the claim, a method that rules out reverse causation or admits it cannot, a species and setting matching the headline, an outcome matching the claim, an effect size in plain units, and a funder and replication status that the reader can find. Failing one item does not make a study wrong. It tells the reader which sentence of the headline to discount, and a study can pass all seven and still be mistaken, or fail several and still be right.
Objections to the seven questions
Is a small sample always a reason to distrust a sleep study?
No. Small samples are normal in lab work that uses polysomnography or PET imaging, because each participant is expensive to measure. What changes is how much weight one result deserves: a 20-person study is a lead to be repeated, while the same finding in three independent labs is a different object. Small samples with large claimed effects and no preregistration deserve the most caution.
Can a sleep tracker settle the self-report problem?
Partly. Wrist actigraphy removes the reliance on memory for total sleep time, and the validation data for that one number is good. It does not remove sampling problems, reverse causation or small effects, and its estimates of wake time and sleep stages are weaker than its estimates of total sleep.
Are industry-funded sleep studies wrong?
Not as a rule. Funding predicts which way results lean on average, not whether a given study is sound. Useful checks are whether the plan was registered before the trial, whether the comparator was a placebo or a real alternative, and whether an independent group has reported the same result.
What is the fastest check when I only have a minute?
Find the sentence in the paper that says how sleep was measured and who was measured. If sleep came from a questionnaire, the finding is about people’s beliefs about their sleep. If the participants were mice or a few dozen adults, the headline probably claims more than the paper does.
Putting the seven questions to work on a headline
Take an invented headline of the kind that circulates each winter: “Sleeping under six hours doubles your dementia risk, study finds.” It is a composite written for this article, not a real report.
Start with question 1 and find the sentence in the paper that says how sleep was measured. If the answer is a questionnaire, the true subject is people’s beliefs about their sleep, and the Lauderdale numbers say short sleepers overstate by more than an hour. Question 3 comes next: were participants followed from healthy baselines, or did early, undiagnosed dementia already shorten their sleep before the study began? Question 5 asks whether the outcome was a diagnosis or a memory-test score, and question 6 asks for the effect in plain units. “Doubles” could mean a rise from 1 in 100 to 2 in 100 over a decade, which is a different claim from 1 in 10 to 2 in 10.
That takes about five minutes with the abstract and the methods section. It will not tell you whether the study is right, only which half of the headline the paper supports. If you track your own sleep while you read, remember that trackers can make sleep worse for people who fixate on the numbers, so swapping a questionnaire for a nightly score trades one measurement problem for another.