Built for AI, or Built for Us?

I read two reports from Anthropic within days of each other this spring, and together they said something neither said alone. The first, Policy on the AI Exponential, opens with a warning aimed at governments: "AI is advancing at exponential speed, and the policymaking process was built for a slower world" (Anthropic, 2026a). The second, When AI Builds Itself, documents the company's own progress toward recursive self-improvement — the point at which an AI system could fully design and develop its successor, with humans supplying little more than the initial goal (Anthropic, 2026b).

Both documents are candid about the stakes. Both are also, worth noting, written by one of the companies setting the pace they describe. That is not a contradiction so much as a position: Anthropic is asking the world to help govern a race it is also running. The honesty is real. So is the race.

The Exponential, By Its Own Account

The numbers in When AI Builds Itself are striking on their own terms. As of May 2026, more than 80% of the code merged into Anthropic’s codebase was written by Claude, up from low single digits before February 2025. On the most open-ended engineering tasks — the kind with no clear specification, where even a skilled person isn’t sure what success looks like — Claude’s success rate reached 76%, up fifty percentage points in six months (Anthropic, 2026b). The report sketches three possible futures: the trend stalls and today’s capabilities simply diffuse outward; AI development becomes substantially automated while humans retain “research taste”; or AI systems become capable of fully recursive self-improvement, with humans shifting toward “oversight, validation, and verification of an expanding virtual lab” (Anthropic, 2026b).

The companion policy paper takes that trajectory as established fact and asks what governments should do about it: require transparency, fund independent evaluators, and grant authority to block deployments that cross catastrophic-risk thresholds (Anthropic, 2026a). It is a serious proposal, and a necessary one. But it is also a proposal written entirely in the grammar of speed — how fast AI is improving, how fast governance must adapt to keep up. What gets less attention, in either report, is a quieter question: improving toward what, and for whom?

When AI Builds Itself — For Whom?

Nearly seventy years ago, Hannah Arendt distinguished between three kinds of human activity: labor, the endless cycle of producing what we consume to stay alive; work, the durable making of a world of things; and action, the speech and deeds through which people appear to one another as who they are (Arendt, 1958). Arendt worried about a society that automates labor without asking what becomes of work and action — a “society of laborers” that has been freed from labor just as labor is the only thing it still knows how to value.

When AI Builds Itself describes something close to that scenario, but for knowledge work. “The doing — i.e., writing the code, running the experiment, producing the result — now costs almost nothing in human time, even if it still has costs in compute,” the report notes, and an Anthropic employee is quoted: “On days where everything works well, I can’t help but think nothing I do matters” (Anthropic, 2026b). That is not a complaint about job loss in the usual sense. It is closer to Arendt’s worry: when the doing is automated, what is left of the human role is direction-setting — and direction-setting, done well, requires exactly the kind of judgment, taste, and lived stake in outcomes that cannot be outsourced to the system being directed.

Racing for AI’s Sake

This is where a second piece of recent scholarship becomes useful, even though it is not, on its face, about AI at all. Susi Geiger’s analysis of tech oligarchs’ “obsession with pronatalism and fertility markets” describes a recurring pattern: an elite circle conjures an existential narrative — population collapse, civilizational extinction — that feels urgent and inevitable, and that narrative then channels enormous capital toward speculative technologies controlled by that same circle (Geiger, 2026). The narrative and the market reinforce each other. Whether the underlying claim is true matters less than whether the belief in it can be sustained long enough for the investments to pay off.

I am not arguing that recursive self-improvement is a hoax — the evidence Anthropic presents is real, and I take it seriously. But the framing of “the exponential” as an autonomous force, something AI is doing to us rather than something a small number of companies and investors are choosing to build, deserves the same scrutiny Geiger applies to fertility markets. “AI building AI” can become its own kind of inevitability narrative: one that makes the current pace feel like physics rather than policy, and that quietly forecloses the question of who gets to slow down, redirect, or say no. Three of the four futures Anthropic itself sketches involve a world reorganized around the needs of an AI development pipeline. Only one of them centers on the people that pipeline is supposed to serve.

What Personal Computing Promised

This is not a new question, even if it is being asked at a new scale. Since the dawn of personal computing, the measure of a technology has never really been how powerful it became. It has been whether that power turned into something helpful, intuitive, and meaningful — for a teacher in an under-resourced school, a small business owner, a retiree, a student whose first language is not the one the interface was designed in. AI is among the most powerful technologies our species has ever built, and with proper care it could unlock real benefit for people everywhere, across every economic circumstance, every culture, every faith. That potential is genuine. But potential is not the same as a guarantee, and a system racing toward self-improvement does not automatically arrive anywhere near “helpful” — it arrives wherever its builders, deliberately or by default, point it.

Toward Society-Centered AI

This is the premise Society & AI was built on, and it has not changed: that truly helpful AI must be centered on people and their needs, not on the trajectory of AI itself. Anthropic’s reports are valuable precisely because they make the trajectory visible — and because they ask, honestly, whether anyone outside the small set of frontier labs gets a meaningful say in where it goes. Our answer is that they must. Society-centered AI research exists to make that “must” concrete: to ask, of every benchmark and every capability gain, not only can it do this but who is this for, and to keep that question in the room for as long as the exponential keeps climbing.

References

Anthropic. (2026a). Policy on the AI exponential. https://www.anthropic.com/policy-on-the-ai-exponential

Anthropic. (2026b). When AI builds itself. The Anthropic Institute. https://www.anthropic.com/institute/recursive-self-improvement

Arendt, H. (1958). The Human Condition. University of Chicago Press. https://archive.org/details/dli.ernet.528547

Geiger, S. (2026). Tech oligarchs’ obsession with pronatalism and fertility markets. Science as Culture. https://doi.org/10.1080/09505431.2026.2639478

Cite this article

Gattupalli, S. (2026). Built for AI, or built for us? Society and AI. https://societyandai.org/perspectives/built-for-ai-or-built-for-us/

Write for Society & AI

Society & AI welcomes scholarly contributions from researchers, educators, librarians, policymakers, and practitioners working at the intersection of artificial intelligence, education, and society. We publish accessible, rigorously argued work that advances public understanding of how AI reshapes institutions, learning, and human experience — and every piece is published free of cost and in open access, in service of education and the public good. If you have a perspective, commentary, or research piece to propose, we would welcome hearing from you: write to the Editorial and Content Director at submissions@societyandai.org.