Silicon Valley Is Downloading Morality It Cannot Compile — and the Weighted Average Will Be Nobody’s Faith
(SeaPRwire) –
There is a quiet protocol negotiation happening in Silicon Valley, and almost nobody is reading the specification. Anthropic co-founder Chris Olah stood at the Vatican earlier this year, at the presentation of Pope Leo XIV’s encyclical on AI, and told the Church that questions of right and wrong “are not questions a lab can answer.” He said traditions like Catholicism have carried them for millennia. OpenAI’s Sam Altman has offered his own version of the same idea. ChatGPT, he said last year, should reflect “that weighted average, or whatever, of humanity’s moral view.” Read those two statements side by side. One treats religion as a deep, living archive. The other treats it as a dataset to be averaged. That gap is the whole story.
The official facts look conscientious. Anthropic says it has met with scholars, clergy, philosophers and ethicists from more than 15 religious and cross-cultural groups, and wants to widen the pool. Meta, Google and Amazon have sent representatives to the Vatican. Microsoft has pursued religious outreach aggressively. Notre Dame philosopher Meghan Sullivan attended an Anthropic session and found it genuinely rigorous — Christians of multiple denominations, technical briefings in layman’s terms, then open debate on everything from whether humans have souls to whether third graders should use chatbots for literacy. Anthropic’s own blog post frames the goal plainly. It is not about aligning Claude with any one tradition. It is about “careful, accumulated thinking on how good character actually forms.” The engagement, by the testimony of participants, is real.
The subtext is where the friction lives. Webb Keane, an anthropologist at the University of Michigan, argues that treating religion as an extractable rule book is a category error with a long pedigree. That view, he notes, is tied to scriptural traditions — and the scriptural religions at the end of the day turn out to be the Abrahamic ones. Train a model on “religious ethics” as a generic layer, and you silently privilege faiths that describe themselves as moral systems while flattening those organized around ritual, doctrine, and relationship. Religious studies scholars go further. Tomoko Masuzawa, author of “The Invention of World Religions,” points out that categories like Buddhism and Hinduism as bounded, comparable systems were not native to their practitioners. They were consolidated by European academics as late as the 19th century. Jonathan Z. Smith made the same point in 1982 — humans have always imagined deities, but “religion” as a discrete, comparable object is a recent Western invention. Silicon Valley has inherited that invention uncritically, because it is convenient. Religion becomes a tidy way of mapping the world and showcasing diversity of thought.
Here is the technical reality underneath the theology. Large language models guess the next word based on patterns absorbed from enormous corpora of human writing, religious texts included. There is no mechanism in that architecture for “equal depth and rigor” across worldviews. There is only statistical weight. Altman’s “weighted average” is not a metaphor — it is a literal description of what the machine does. And a weighted average of contradictory moral traditions is not a synthesis. It is a distortion field in which the most textually prolific, most rule-articulated traditions dominate, and everyone else gets smoothed into noise.
The warning signs are already visible, and the honest participants know it. Paolo Carozza, a member of the independent oversight apparatus surrounding this work, was among those pressing for sharper definitions of what companies actually intend to extract. The concern is structural, not rhetorical. Once moral authority is routed through a handful of proprietary models, the model vendor becomes the de facto standards body for global ethics — one with no charter, no ratification process, and no appeal mechanism. The internet balkanized along national lines over two decades. Moral computation risks balkanizing along corporate lines in a fraction of that time, with each foundation model quietly encoding a different flattened theology.
If Anthropic, OpenAI, and their peers want religious input to mean anything, they should publish their consultation taxonomies, name the traditions weighted into training, and submit the aggregation method to external audit. Otherwise the “weighted average of humanity’s moral view” will resolve, by default, to the view of whoever wrote the most text in English.