ARION's proposal builds on independently published research across bioacoustics, NLP, and continual pretraining. Here are the foundations.
Sperm whale codas operate as a combinatorial coding system with a phonetic alphabet based on rhythm, tempo, rubato, and ornamentation — far richer than previously known.
doi.org/10.1038/s41467-024-47221-8Sperm whale clicks contain vowel-like and diphthong-like spectral qualities, analyzed via source-filter theory from human speech — expanding the phonetic alphabet.
Multilingual models spontaneously align semantically equivalent concepts across languages without any parallel data — even when there is no shared vocabulary.
aclanthology.org/2020.acl-main.536/Pretraining on 100 languages at scale produces strong cross-lingual transfer, especially for low-resource languages like Swahili and Urdu.
aclanthology.org/2020.acl-main.747/First generative model for dolphin vocalizations. Predicts and generates realistic whistles, clicks, and burst pulses. Trained on 40+ years of Atlantic spotted dolphin recordings.
deepmind.google/models/gemma/dolphingemma/Transformer-based pipeline that automatically detects, segments, and annotates sperm whale codas using the phonetic alphabet. Runs on public datasets.
github.com/ceti-aiAdding small volumes of novel tokens during continual pretraining yields gains in the target domain with negligible regression on high-resource benchmarks.
Demonstrated the iterative data-cleaning loop: noisy web text → model → cleaner data → better model. The same bootstrapping pattern ARION proposes for whale data.
Cross-lingual alignments strengthen with scale and can be induced by targeted interventions, including the addition of previously unseen languages.
Shane Gero founds the Dominica Sperm Whale Project — the world's longest-running longitudinal study of sperm whale social structure and communication.
GPT-2 demonstrates that language models can learn coherent, structured knowledge from raw noisy web text at scale — establishing the bootstrapping loop.
Conneau et al. prove emergent cross-lingual alignment without parallel data. GPT-3 demonstrates iterative data cleaning. Project CETI founded as a TED Audacious Project.
Sharma et al. (Nature Communications) discover that sperm whale codas constitute a combinatorial phonetic alphabet — rhythm, tempo, rubato, ornamentation, vowel quality.
Beguš et al. identify vowel-like spectral patterns in whale clicks. DolphinGemma announced by Google DeepMind. WhAM open-sourced by Project CETI. Nag et al. validate continual pretraining safety.
ARION founded. The founding paper connects these threads and proposes the first concrete pipeline for interspecies language alignment in frontier model pretraining.