Chapter 24 - Cross-Modal Pattern Hunting
In Shark Bay, Western Australia, a population of bottlenose dolphins does something no other known dolphin population does anywhere else in the world: some individuals - overwhelmingly females, overwhelmingly passed from mother to daughter - carry a marine sponge on their snout while foraging along the seafloor, using it as a glove to protect against abrasion and the spines of buried fish while they root through the substrate for prey too well camouflaged for echolocation alone to find. This behavior, "sponging," has been studied for decades and is one of the best-documented examples of tool use and cultural transmission in a non-primate species. It is also a perfect illustration of a specific, underexploited opportunity for morphic resonance research, because a single instance of sponging behavior doesn't exist in the research record as one type of data. It exists as several: video footage of the behavior itself, genetic samples establishing which dolphins are related to which, GPS tracking data showing the geographic distribution of spongers within the bay, environmental data on seafloor composition where sponging occurs most, and decades of field-naturalist notes documenting the behavior's spread through the population's social network.
Historically, each of these data types gets analyzed separately, by researchers with different specializations, using different tools, often published in different papers years apart - the geneticists tracking maternal lineage, the behavioral ecologists tracking social transmission, the naturalists writing up the qualitative field notes - with any connections between the four made informally, by a human researcher's memory and judgment, rather than through any systematic, joint computational analysis. This chapter is about the machine learning discipline specifically designed to break down that separation: cross-modal learning, and why it may be one of the most promising, currently underexploited tools available for morphic resonance research. We'll build the whole chapter around the Shark Bay case, moving from general principle to specific, worked application and back, rather than treating it as one example among many.
What "Modality" Means, and Why Joining Them Is Hard
In machine learning, a modality refers to a fundamentally different type or format of data - text, images, audio, video, tabular numerical data, genomic sequences, and time-series data are all distinct modalities, each with its own characteristic statistical structure, and historically, each requiring substantially different specialized model architectures to process effectively. The technical challenge of cross-modal learning is building a single system, or a coordinated set of systems, capable of finding meaningful correlational structure that spans multiple modalities simultaneously - recognizing that a specific pattern in Shark Bay's genomic data, a specific pattern in its behavioral time-series data, and a specific pattern in decades of field-naturalist text descriptions are all, in some rigorous statistical sense, describing the same underlying phenomenon, even though the three data types have entirely different native mathematical structure and entirely different expert communities historically studying them.
The key modern technical innovation making this tractable is the shared embedding space: rather than trying to directly compare raw pixel values from sponging-behavior video to raw genomic sequence data, which have no natural basis for direct comparison, modern cross-modal systems train separate encoder models for each modality, each producing a numerical vector representation - an "embedding" - in a common, shared mathematical space, with the training process, often using the contrastive learning technique from Chapter 21, specifically designed to ensure that embeddings representing "the same underlying thing," across different modalities, end up close together in this shared space. This is exactly the underlying technical principle behind modern multimodal AI systems capable of matching a text description to a relevant image - the same core technology, redirected at scientific data rather than internet images and captions.
Building the Shark Bay Pipeline, Step by Step
Here is what jointly embedding the Shark Bay case would actually involve, stated concretely enough to be a real project outline rather than an abstraction. First, an encoder for the accelerometer and video-derived movement data, trained to classify the specific motor pattern of sponge-foraging against the population's other documented foraging techniques - a well-defined, supervised sub-task, since decades of labeled field footage already exist. Second, an encoder for the genomic and pedigree data, trained to produce a representation of each individual's maternal lineage and relatedness to known spongers, using standard population-genetics methods. Third, an encoder for the environmental data - seafloor depth, substrate type, prey density - at each documented sponging location. Fourth, and this is the piece that makes the whole exercise genuinely novel rather than a conventional multi-variable regression, a language-model-based encoder, of exactly the kind built in Chapter 20, trained on the full digitized corpus of field-naturalist notes and published behavioral ecology literature describing sponging and functionally similar tool-use behaviors documented in other species and other locations worldwide - chimpanzee termite-fishing, New Caledonian crow hook-tool manufacture, sea otter rock-anvil use - projected into the same shared embedding space as the first three.
Once all four encoders are trained and jointly aligned, the system can answer a question no single-modality analysis could pose cleanly: does sponging's pattern of emergence and spread within Shark Bay's own dolphin population - which the genetic and behavioral data show clearly follows maternal lineage, an entirely conventional, well-established cultural-transmission pathway - show any residual similarity, in the shared embedding space, to functionally analogous tool-use innovations in unrelated, geographically distant species and populations, beyond what the environmental and ecological similarity between the two situations would already predict? This reframes the old "independent invention versus resonance" question from Chapter 8's discussion of Imo the macaque in fully cross-modal, quantitatively rigorous terms - allowing a system to flag cases of behavioral similarity that a researcher looking at only movement data, or only genetic data, or only historical text records, in isolation, would have no way of noticing, precisely because Sheldrake's own hypothesis, established carefully in Chapter 3, proposes that resonance operates on the basis of overall pattern similarity, not any single measurable variable.
What the Null Model Has to Rule Out First
Before any residual similarity in the Shark Bay pipeline's shared embedding space could count as interesting, it has to survive a null model at least as demanding as the ones built elsewhere in this Part. The most obvious confound is convergent ecological pressure: tool-assisted foraging for buried, well-camouflaged prey is a solution multiple marine and terrestrial species have arrived at independently, wherever the relevant prey and substrate conditions co-occur, for reasons that have nothing to do with cross-population resonance and everything to do with ordinary natural selection acting on similar problems. A properly built null model needs its own encoder for exactly this environmental-pressure variable - prey camouflage difficulty, substrate type, foraging energy cost - trained to predict tool-use likelihood from ecological conditions alone, with no access to any of the other three modalities. Only a residual similarity that survives being compared against that baseline, not just against the raw environmental variables entered as regression covariates, deserves to be called anomalous. This is a meaningfully harder bar than the pipeline's basic validation step described above, and it is the specific place where a graduate student attempting this project is most likely to underbuild the null model and overstate a finding - a caution worth taking as seriously as the negative-control discipline Chapter 27 develops in full.
It's also worth being honest about data provenance and cost here, since "decades of documented behavioral ecology data" undersells the real engineering lift involved. The Shark Bay dolphin research program itself represents one of the longest-running individual-based marine mammal studies in the world, with genetic sampling, behavioral observation, and photo-identification records spanning back to the 1980s - a genuine asset for this proposal, but one accumulated by a specific research group over four decades, not a ready-made public dataset. A realistic version of this project requires a formal collaboration with the researchers who hold that data, not a download. The field-naturalist text corpus for cross-species tool-use comparison is more readily assembled from published behavioral ecology literature, much of which is available through standard academic access, but digitizing and structuring decades of primary field notes specifically, as opposed to their published summaries, is itself a nontrivial data-engineering task, and a realistic budget for the full four-encoder pipeline, including the collaboration-building time this kind of partnership requires, should assume twelve to eighteen months before any anomaly-detection analysis actually begins.
A Second Application: Genomic and Behavioral Co-Analysis
A second, more specifically biological application of cross-modal methods deserves direct attention, because it bears directly on the morphogenesis and epigenetics material from Chapters 11 and 14: jointly analyzing genomic sequence and epigenetic marker data alongside behavioral and developmental outcome data, across large populations, using cross-modal models specifically designed to identify cases where behavioral or developmental similarity between individuals substantially exceeds what their genomic and known-epigenetic similarity alone would predict - precisely the kind of residual, unexplained correlational structure that would need careful further investigation.
This is methodologically demanding and requires real statistical care: genomic and epigenetic data are extraordinarily high-dimensional, meaning the multiple-comparisons and curse-of-dimensionality problems introduced in Chapter 21 apply here with particular force, and any claimed residual correlational structure needs to survive extremely conservative statistical correction before being treated as more than a preliminary, hypothesis-generating observation. But the underlying methodological opportunity is real and, as far as I'm aware, has not yet been systematically pursued specifically in service of morphic resonance research, despite large, publicly available genomic and phenotypic datasets already existing for entirely separate, mainstream medical and biological research purposes.
What Validation Actually Looks Like
Before any of this counts as evidence rather than an interesting demonstration, the pipeline needs to earn trust the way every other method in this Part has: by first succeeding at a task where the answer is already known. For the Shark Bay pipeline specifically, that means running the full four-encoder system on the historical record with the "target" case deliberately withheld, and checking whether it correctly recovers the known relationships researchers have already established through decades of manual work - does it correctly cluster sponging with maternal lineage rather than with, say, geographic proximity alone, which we know from the existing genetic literature is the wrong grouping? Does it correctly distinguish sponging from superficially similar but functionally different tool-use behaviors documented elsewhere, rather than conflating anything involving an object held in the mouth? Only once the pipeline reliably reconstructs the settled science should it be trusted to flag genuinely novel candidate similarities, exactly following the negative-control discipline Chapter 27 will insist on as mandatory practice for every method in this Part, cross-modal pipelines very much included. A pipeline that can't recover Imo's own well-documented case from Chapter 8 has no business being pointed at an undocumented one.
The Interpretability Challenge
Cross-modal systems, precisely because of their power to find subtle, high-dimensional joint patterns that no human analyst could identify by inspection, raise a distinctive methodological challenge: interpretability. When the Shark Bay pipeline flags a similarity between the local emergence of sponging and a tool-use innovation on another continent, it is not always straightforward to determine, after the fact, exactly which specific features of the original, raw multi-modal data actually drove that similarity judgment - a well-documented general challenge in modern deep learning, sometimes called the "black box" problem, and one with particular importance here, because a claimed anomalous similarity that cannot be traced back to specific, interpretable, checkable features is far more vulnerable to exactly the "unfalsifiable in practice" criticism Chapter 6 identified as this field's central historical weakness. A rigorous research program using cross-modal methods has an obligation to invest in the growing subfield of explainable AI - attention visualization, feature attribution methods such as SHAP and integrated gradients, and concept-activation-vector methods that attempt to identify which human-interpretable concepts a model's internal representations correspond to - not as an optional add-on, but as a core, non-negotiable part of the methodology.
Concretely, for the Shark Bay pipeline, this means that a flagged similarity should come packaged with an answer to a specific question: which of the four input modalities actually drove the flag, and can that driving feature be independently verified by a human expert without the model's help? If a flagged similarity turns out to be driven primarily by the environmental encoder - both tool-use populations happen to forage over similarly structured seafloor - that is informative, but it is a finding about convergent ecology, not about anything resonance-relevant, and the interpretability step is what tells you which of those two very different papers you are actually writing.
Questions for the Curious
Design a cross-modal embedding pipeline joining movement and behavioral sensor data with digitized historical field-naturalist text records, using the Shark Bay sponging case or a comparably well-documented species with known behavioral innovations. Validate the pipeline by testing whether it correctly identifies already-known behavioral analogues - chimpanzee tool use, corvid tool use - before searching for undocumented ones.
Using a publicly available model-organism genomic and phenotypic dataset, construct a cross-modal analysis identifying pairs or clusters of individuals with behavioral or developmental similarity exceeding what genomic and known epigenetic markers predict. Apply rigorous multiple-comparisons correction and propose a specific, targeted follow-up wet-lab experiment for the most robust identified anomaly.
Compare at least two explainable-AI techniques - attention visualization versus feature attribution, for instance - applied to the same cross-modal similarity model, and assess how consistently they identify the same underlying features as responsible for a given flagged similarity, as a direct test of how reliable current interpretability methods actually are for this kind of scientific application.
Propose a formal, quantitative definition of "morphic unit similarity," extending the qualitative concept from Chapter 3, grounded in a specific, fully specified cross-modal embedding architecture, and discuss what would be lost, philosophically and scientifically, in reducing Sheldrake's original, more holistic qualitative concept to this kind of specific, computationally tractable operationalization.
Looking Forward
We've now covered five major technological approaches - language models, unsupervised pattern discovery, simulation and digital twins, sensor networks, and cross-modal learning - each offering a genuine, non-hyped, technically grounded contribution to a modern morphic resonance research program. Chapter 25 pulls one specific thread from Chapters 20 and 21 together and examines it in much greater depth: the provocative and genuinely debated question of whether large language models' own internal representations might themselves constitute a novel, if indirect, kind of instrument for detecting resonance-like structure in the vast cultural record they're trained on.