How to Check a Sleep or Habit Claim in Six Minutes
Six one-minute checks for judging whether a sleep or habit study is reliable: comparison group, assignment, measurement, denominator, interval, and the conclusion.
In this article12 sections
A paper titled “Alarm Clock-Based Reminder for Improving Low Adherence on Option B Plus Antiretroviral Therapy” sounds like proof that clocks help mothers take HIV medicine. Nobody in it was given a clock.
A sleep or habit claim deserves trust when the study compared like with like, assigned the habit instead of letting people pick it, measured something harder to flatter than a survey answer, kept most of its volunteers, and reported a range of plausible effects. Six checks cover that.
DontSnooze, the app behind this blog, bets that friends seeing your proof makes you wake up; run the steps on that claim and it fails step 2, because no randomised trial has tested it, and the nearest evidence is on police body cameras.
1. Find who was compared with whom
The Ethiopian survey (a 2020 paper whose table of reminder types is easy to misread) sorted 350 mothers by reminder type: 293 used a clock, 57 relied on friends or parents. About 71% of the first group reported good adherence against about 49% of the second. That is clock users versus people who depend on others, not clocks versus no clocks.
2. Ask who decided
Clock owners may just be the organised ones; assignment fixes that. In Lester et al. (The Lancet, 2010), 538 Kenyan adults starting HIV treatment were randomly given weekly nurse texts or standard care. Adherence above 95% was reported by 168 of 273 (61.5%) versus 132 of 265 (49.8%). (Abstract only.)
3. Check what was measured
The Ethiopian outcome was four self-reported questions. Lester’s team added viral load, which a friendly text cannot talk into shape.
4. Count the denominator
Lally’s UCL habit study recruited 96 volunteers. The famous 66 days is the median among the 39 whose curves fitted well (as a close reading of the paper’s range reports it), with individual times from 18 to 254 days.
5. Demand the interval
“Significant” is a yes-or-no. The Ethiopian odds ratio was 2.51 with a 95% interval of 1.30 to 4.86; the top is 3.7 times the bottom (4.86 / 1.30). It holds a modest effect and a big one.
6. Hold the conclusion against the study type
The same abstract urges providers to use “a clock as a reminder,” after a one-time survey that handed out no clocks. Then check the end of the paper for funders; Lester’s names the US President’s Emergency Plan for AIDS Relief.
Copy-paste scorecard
- Comparison: fair control, or a different kind of person?
- Assignment: randomised, or self-chosen?
- Measure: checkable outcome, or self-report only?
- Denominator: most starters counted?
- Size: effect with an interval, not just “significant”?
- Fit: conclusion no bigger than the study?
One point per pass. Under three, treat the claim as a hypothesis. That cut-off is my own rule of thumb.
What the checklist cannot do
It grades credibility, not truth. A study can pass all six and find nothing: the 2023 WelTel PMTCT trial in Kenya randomised 600 pregnant women to weekly texts or standard care, and retention in care was 210/299 versus 207/301 (risk ratio 1.02, interval 0.92 to 1.14). It also cannot catch fabricated data.
When the checks pull in different directions
Can the six checks be done from the abstract alone?
Mostly: abstracts usually cover checks 2, 4, 5 and 6. Check 1 often needs a results table, and an abstract that hides its comparison group earns a point against.
Does “randomised” end the argument?
No. In the 2010 Kenyan texting trial participants knew their group, and its number needed to treat for adherence, 9, carried an interval of 5.0 to 29.5.
How wide is too wide for an interval?
Wide enough that its two ends would lead to different decisions. Odds ratios of 1.30 and 4.86 would, so a range spanning both settles little.