How we read your data
Are your fitness app's insights real?
Search enough health data and something always looks meaningful by pure luck. Most apps never check. Here is the arithmetic of why, measured on our own engine, and the three questions worth asking of anything that shows you a pattern.
Your watch records a lot about you. Heart rate variability overnight, resting heart rate, sleep stages, breathing rate, wrist temperature, steps, every workout. An app that wants to tell you something interesting has an easy job: line up any two of those, split your days into the ones where something happened and the ones where it did not, and compare.
Do that once and you are doing science. Do it two hundred times and you are shopping.
The problem has a name
It is called the multiple comparisons problem, and it is not subtle. A significance test set at the usual 5% threshold is, by construction, willing to be wrong 5% of the time. That is what the number means: if nothing whatsoever is going on, roughly one test in twenty will still look convincing.
One test in twenty is fine when you run one test. It is a disaster when you run a catalogue. Ask twenty questions of pure random noise and you should expect one answer to come back looking real. Ask a hundred and you should expect five. The test is not broken. It is doing exactly what it promised. The mistake is asking it a hundred questions and then reporting only the winners.
This is why "we found 13 patterns in your data" is a red flag rather than a feature. The more an app looks, the more it will find, and the number it shows you tells you nothing about whether any of it is true.
We measured it on ourselves
It is easy to assert this and harder to put a number on it, so we did. We generated 300 simulated runs of 40 days of pure noise: data with no real relationships in it at all, by construction. Then we ran our own hypothesis catalogue over each one and counted how often it surfaced something.
We also planted a genuine effect in a separate set of runs, because a filter that suppresses everything is not honest, it is just quiet. Both numbers matter.
| Correction | Noise runs that surfaced a finding | Strong effect found | Weak effect found |
|---|---|---|---|
| None | 35.0% | 38 / 40 | 8 / 40 |
| FDR at q = 0.20 | 18.7% | 38 / 40 | 8 / 40 |
| FDR at q = 0.10 | 11.3% | 38 / 40 | 8 / 40 |
| FDR at q = 0.05 | 6.3% | 37 / 40 | 5 / 40 |
of runs on pure noise produced a finding that passed an ordinary significance test. More than one in three. Every one of them would have looked, to a reader, exactly like a real discovery about their body.
Read the first row again, because it is the whole argument. With no correction applied, an engine like this hands you a confident, specific, personalised claim about your own health on more than a third of runs where the underlying data is meaningless. Nothing about how that claim is worded would tip you off. It has a number in it, a sample size, and a plausible mechanism you can imagine.
What a correction actually does
The fix is not to test less. It is to raise the bar in proportion to how much you looked. We use a Benjamini-Hochberg step-up procedure, which controls the false discovery rate: not the chance that any single test is wrong, but the share of the findings you are shown that are expected to be false alarms.
The mechanics are less important than the consequence. Every hypothesis in the catalogue gets its p-value. They are sorted. Then the threshold each one must clear depends on how many were tested in total, so a finding that would sail through on its own has to work harder when it is one of two hundred. Findings that do not clear the adjusted bar are discarded.
We set q at 0.10, and the table above is why. It removes roughly two thirds of the noise at no cost to detection: the strong planted effect is still found 38 times out of 40, the same as with no correction at all. Tightening further to 0.05 buys a little more noise reduction but starts losing real findings in the weak-to-moderate band, which is exactly the band worth surfacing. Nothing here is free, and pretending otherwise would be its own kind of dishonesty.
Correction is necessary, not sufficient
A p-value answers one narrow question: how surprising is this difference if nothing is going on. It does not tell you the difference is large enough to matter. HRV routinely swings 10% on nothing at all, while resting heart rate barely moves 4% unless something real is happening. A statistically clean 1% shift in either is a true fact and a useless one.
So an effect-size floor has to sit alongside the correction, and it has to be set per signal rather than globally, because the natural volatility of each measurement is different. Steps are wild. Resting heart rate is stubborn. One threshold across all of them is either too loose for the noisy signals or too strict for the quiet ones.
There is a floor on sample size too. Five paired days in each arm is our minimum, and it is a minimum, not a target. Comparisons drawn from three nights against three nights are arithmetic, not evidence.
Three questions to ask any app
- 01 How many things did you test? If an app cannot tell you, it is not tracking the denominator, which means it cannot be correcting for it. The count of questions asked is the single most important number behind any finding, and it is almost never shown.
- 02 What did you throw away? An engine that reports only what it found is indistinguishable from an engine that reports whatever it found first. The discard count is the honest signal, and it should usually be much larger than the keep count.
- 03 Is this correlation or cause? Observational data from your own life is self-selected. You chose the late nights and the rest days, and whatever made you choose them may be doing the work. "On days after late meals your HRV was 8% lower" is a defensible claim. "Late meals lower your HRV" is not the same sentence.
What we do about it
Every finding in Tensile clears three bars before you see it: at least five paired days in each arm, an effect large enough to matter for that specific signal, and a Welch's unequal-variance t-test at 5%. Whatever survives all three is then put through Benjamini-Hochberg at q = 0.10, against the count of hypotheses that had enough data to be tested at all rather than a flattering subset.
What fails is deleted. Not greyed out, not filed under "emerging trends", not shown with a smaller font. Deleted. And the count of what we deleted is shown to you, because the discard number is the part that tells you the keep number means something.
In practice that means the app frequently has less to say than it could. A typical run might test thirteen ideas about you and keep two. Somewhere in the eleven we threw away was probably something true that we could not prove yet, and it will surface later when there is more data behind it. That is the trade, and we would rather make it in that direction.
None of this makes a finding medical advice, and none of it makes a correlation a cause. It only means that when Tensile tells you something about your body, the number of other things it looked at first has already been paid for.
More in Learn. Related: how the app reads your history on day one, and what happens to your health data.