Chapter 12 - Statistics, Replication, and the File Drawer

In 2015, a consortium of over 270 psychologists published the results of an enormous, coordinated effort called the Reproducibility Project, in which they attempted to directly replicate one hundred published studies from top psychology journals, using, wherever possible, the original materials and larger sample sizes than the original studies had used. Only about 36 to 39 percent of the replications (depending on exactly how "successful replication" was defined) produced a statistically significant effect in the same direction as the original study. This was not a study of parapsychology, or of morphic resonance, or of any fringe science whatsoever - it was a study of mainstream, thoroughly conventional, peer-reviewed experimental psychology, the kind of research published in the most respected journals in the field, conducted by researchers with no particular ideological stake in any unconventional hypothesis. And the great majority of its findings failed to replicate.

This chapter exists because you cannot fairly evaluate any of the evidence discussed so far in Part II - the rats, the staring, the crystals - without understanding the broader statistical and methodological crisis that this 2015 study crystallized into public awareness, a crisis that has forced an entire generation of scientists across many fields to fundamentally rethink how much weight any single published, statistically significant finding actually deserves. Understanding this crisis properly will make you a sharper reader of morphic resonance evidence specifically, and a sharper reader of scientific claims generally, for the rest of your career.

What a P-Value Actually Means (and Doesn't)

Let's build this from the ground up, in plain language first, because the single most consequential statistical misunderstanding in all of contested science is a misreading of what a p-value actually tells you.

A p-value answers a narrow, specific question: assuming there is truly no real effect (the "null hypothesis" is true - for instance, assuming morphic resonance genuinely has no influence on crystal formation rates, or on staring detection, or on rat learning speed), what is the probability of observing a result at least as extreme as the one actually obtained, purely by chance? A p-value of 0.05 (the conventional, if somewhat arbitrary, threshold for "statistical significance" across much of science) means that, if there were truly no real effect, a result this extreme or more so would occur by chance alone about one time in twenty.

Here is the crucial thing a p-value does not tell you, and that a shocking number of working scientists, not just laypeople, routinely misunderstand: it does not tell you the probability that the null hypothesis is true, given your data. It does not tell you the probability that your finding will replicate. And critically, it does not, on its own, tell you the probability that a real, non-chance effect actually exists, because that depends heavily on how plausible the hypothesis was before you ran the experiment (its "prior probability," in the language of Bayesian statistics we'll formalize properly in a moment) and on how many other researchers tried similar experiments and got null results that never got published. A p-value of 0.05, applied to a wildly implausible hypothesis, tested by many different research groups where only the rare "significant" result gets published, can be entirely consistent with the null hypothesis being true essentially all of the time, in the actual population of real-world attempts - a point statisticians call the base rate fallacy, and it is, I would argue, the single most important statistical concept for evaluating any minority scientific claim, morphic resonance very much included.

The File Drawer Problem, Quantified

We introduced this concept informally in Chapter 6; let's now make it fully precise, because the mathematics of it are genuinely illuminating, not just a vague gesture at "bias."

Imagine, purely as a thought experiment, that morphic resonance has absolutely no real effect on anything - the null hypothesis is entirely true across the board. Now imagine one hundred independent research teams around the world, each testing some version of the hypothesis (staring detection, crystal formation, maze learning), each using the conventional 0.05 significance threshold. By the very definition of a p-value, we should expect roughly five of those one hundred teams - purely by chance, with no real effect operating at all - to obtain a "statistically significant" positive result. If those five teams are the ones who go on to publish (because a null result is, as we discussed, less exciting, less publishable, and less career-advancing than a positive one, an asymmetry that is well-documented across essentially all of science, not just fringe topics), and the ninety-five teams who found nothing quietly move on to other projects without publishing, then the published literature on this topic will show a 100% positive result rate, entirely consistent with a real effect, despite there being no real effect operating anywhere in the underlying population of actual experiments conducted. This is the file-drawer problem in its starkest, purely quantitative form, and it is a mathematical inevitability, not a moral accusation against any individual researcher's honesty - it requires no fraud, no bad faith, and no conscious manipulation of data whatsoever, only the ordinary, universally human tendency to be more motivated to write up and submit an exciting positive finding than a deflating null one.

Meta-analysis - the statistical technique of formally combining the results of multiple studies on the same question into a single, pooled estimate - was developed precisely to try to address this problem, and modern meta-analytic methods include specific statistical tools (funnel plots, which look for the characteristic asymmetric scatter pattern that publication bias produces; trim-and-fill methods, which attempt to estimate how many "missing" null studies would need to exist to produce the observed asymmetry) designed to detect and, to some extent, correct for file-drawer distortion.

Several meta-analyses of the parapsychology-adjacent literature relevant to morphic resonance - including work on the staring-detection paradigm we discussed in Chapter 9 - have been published using exactly these techniques, and the honest state of that specific meta-analytic literature is genuinely contested: some analyses suggest the pooled effect, while small, survives reasonable correction for publication bias; others suggest the effect shrinks toward zero once file-drawer correction is properly applied. This is not a cop-out non-answer; it is a precise description of where the actual statistical literature currently sits, and it should calibrate your confidence accordingly - modest evidence, of a small effect, whose robustness to publication bias correction is itself a live, technical, ongoing dispute among statisticians who specialize in meta-analysis, not settled either way.

Researcher Degrees of Freedom and P-Hacking

There is a second, related statistical pathology worth understanding in full technical detail, because it recurs constantly in exactly the kind of behavioral and biological data relevant to morphic resonance research: what statisticians call researcher degrees of freedom, more colloquially known as p-hacking.

Any real experiment involves a surprising number of small, seemingly innocuous analytical choices that are not always specified in advance: which specific outlier-exclusion criteria to apply, whether to analyze the data with or without a handful of ambiguous or partial trials, which of several plausible statistical tests to use, whether to check for the effect within specific subgroups (only trials where the starer was highly motivated; only rats tested in the morning; only crystals grown at a specific temperature) if the overall analysis doesn't reach significance. A landmark 2011 paper by the psychologists Joseph Simmons, Leif Nelson, and Uri Simonsohn, provocatively titled "False-Positive Psychology," demonstrated mathematically and empirically that a researcher who has even a modest number of these small, defensible-sounding analytical choices available, and who is not strictly bound by a pre-specified analysis plan, can dramatically inflate their real, effective false-positive rate - sometimes from the nominal 5% up to 40% or more - even while every single individual choice they make, examined in isolation, seems like an entirely reasonable, good-faith judgment call, with zero deliberate fraud involved at any single step.

This matters enormously for the McDougall rat data, the staring experiments, and the crystal formation anecdotes we've covered across this Part, because a great deal of that historical evidence was generated and analyzed before the discipline of pre-registration (publicly, verifiably committing to your exact hypothesis, sample size, and analysis plan before collecting any data, specifically to close off researcher degrees of freedom after the fact) became a widely adopted methodological norm, which happened across most of experimental psychology only in roughly the past decade, substantially after most of the classic morphic resonance literature was already published. This is not a claim that any specific researcher we've discussed acted in bad faith; it's a structural, field-wide observation that applies to essentially all pre-2010s experimental psychology and biology, mainstream and fringe alike, and that should make you appropriately more cautious about any single historical finding's precise reported significance level, while not requiring you to throw out the entire body of evidence as worthless.

A Bayesian Corrective

I promised a Bayesian formalization of the base-rate point above; let's deliver it, because it's genuinely useful and not as intimidating as it sounds. Bayesian statistics, in contrast to the more traditional "frequentist" p-value framework we've used so far, explicitly incorporates your prior belief about a hypothesis's plausibility, updates that prior in light of new evidence, and produces a posterior belief that reflects both.

Concretely: if you consider morphic resonance, before seeing any specific experimental result, to be a fairly implausible hypothesis - a low "prior probability," reflecting its tension with well-established physics, examined rigorously in Chapter 16, and the absence of any accepted mechanism - then even a study reporting a conventionally "significant" p-value of 0.05 should, on a proper Bayesian analysis, move your belief only modestly, because the same data would also be reasonably likely to occur under mundane explanations that themselves have much higher prior plausibility.

Here is the arithmetic behind that claim, worked through with round numbers so you can see exactly where it comes from. Suppose, before seeing any data, you assign morphic resonance a prior probability of 1 in 1,000 of being true for a specific claim - a deliberately harsh but not unreasonable prior, given the physics tension established in Part III. Suppose a study returns a result significant at p = 0.05, and suppose, generously, that if morphic resonance were true, a study like this one would detect it correctly 80% of the time (its "statistical power"). Bayes' theorem tells you how to combine these numbers into a posterior probability - your updated belief after seeing the result:

$$P(\text{true} \mid \text{significant}) = \frac{P(\text{significant} \mid \text{true}) \times P(\text{true})}{P(\text{significant})}$$

Plugging in: the numerator is 0.80 × 0.001 = 0.0008. The denominator - the total probability of seeing a significant result, whether the hypothesis is true or false - is that same 0.0008, plus the probability of a false positive: 0.05 (your significance threshold) × 0.999 (the prior probability the hypothesis is false) ≈ 0.05. Adding those gives roughly 0.0508. Divide: 0.0008 / 0.0508 ≈ 0.016. A single "significant" result, under these assumptions, moves your belief from 1 in 1,000 to roughly 1.6 in 100 - a real update, fifteen-fold, and still a long way from confidence. This is not special pleading against morphic resonance specifically; it is exactly the same arithmetic that should make any careful scientist skeptical of a single significant result for a claim requiring the physics of Chapter 16 to be substantially revised, versus an equally significant result for a claim already consistent with existing theory, where the prior might be 1 in 10 rather than 1 in 1,000, and the same p = 0.05 result would move you to roughly 64% confidence in a single study. The phrase "extraordinary claims require extraordinarily strong evidence" is not a rhetorical dismissal. It is what this equation says, worked through with real numbers, and it's worth keeping as a permanent tool in your own scientific reasoning, applicable to fringe science and mainstream science alike.

The Replication Crisis Beyond Psychology

It's worth being clear that none of this is unique to parapsychology-adjacent research, and understanding that fact properly should discourage you from concluding "well, this just shows morphic resonance research is uniquely unreliable." The 2015 Reproducibility Project result cited at the top of this chapter concerned entirely mainstream cognitive and social psychology. Similar, though generally somewhat less severe, replication concerns have subsequently been documented in cancer biology (a widely cited 2012 analysis by researchers at the biotechnology company Amgen reported that they were able to confirm the findings of only 6 out of 53 "landmark" preclinical cancer studies they attempted to replicate internally), in economics, and in numerous other fields far removed from anything touching morphic resonance. The point of bringing this up is not "everything is equally unreliable, so nothing matters" - that would be a lazy, unhelpful nihilism. The point is that the specific statistical pathologies making morphic resonance evidence hard to evaluate - publication bias, researcher degrees of freedom, small sample sizes, the seductive but misleading precision of a single p-value - are general features of how science was conducted, across nearly every field, before the recent, ongoing push toward pre-registration, larger sample sizes, and open data. This general methodological reckoning is, in fact, wonderful news for anyone hoping to finally get a clean answer about morphic resonance specifically. The tools developed to fix mainstream science's replication crisis - pre-registration platforms, open-data requirements, large-scale collaborative replication consortia, and, as we'll see beginning in Chapter 20, computational and machine-learning methods for detecting and correcting exactly these biases at scale - are directly, immediately applicable to finally running the kind of decisive, well-powered morphic resonance experiments that four decades of smaller, less rigorous studies have failed to deliver.

Looking Forward

We now have the full statistical toolkit needed to evaluate any claim in this space with genuine rigor: what a p-value can and cannot tell you, how the file-drawer problem can manufacture an entirely illusory positive literature out of a true null effect with zero fraud involved, how researcher degrees of freedom can inflate false-positive rates even among well-intentioned scientists, and how a Bayesian framing lets you properly weigh a study's result against the hypothesis's prior plausibility rather than treating every significant p-value as equally meaningful regardless of context.

Chapter 13 closes out Part II by turning this toolkit directly onto the single most important remaining question: what would it actually take, in concrete and specific terms, to change a well-calibrated skeptic's mind about morphic resonance - and, symmetrically, what would it take to change a well-calibrated believer's mind against it? We'll meet Sheldrake's most serious critics directly, engage with their strongest arguments on their own terms, and close Part II with a genuinely balanced accounting of where the evidentiary chips currently lie, before we turn, in Part III, to the harder theoretical question of whether any known or plausible physics could accommodate the hypothesis at all.