Towards a Science of Capable Autonomous Agents & Provable Safety

As AI agents swiftly become increasingly capable and ubiquitous throughout society, it becomes a more pressing matter to understand (1) what we might expect to emerge inside of them, and (2) what, if any, guarantees we may have in controlling them.

My long-term research vision is to build a natural science of autonomous intelligence: executable theories of how agents generate useful behavior in open worlds, and how those mechanisms can remain under human control. In Part I of this series, I will start with what capable agents must necessarily have inside of them. Which properties of their internals must necessarily follow from observing their behavior of doing a task well, even if we did not explicitly put them into the architecture?

— Part I

— Part II

#trust

Research acceleration: The jagged frontier

Over the course of this year, AI systems have become increasingly capable, potentially increasing their ability to contribute to AI research and development themselves. Today, we’re providing an update on the state of research acceleration at OpenAI, based on internal measurements of how coding agents have accelerated and changed the AI development cycle.

… Since our first report⁠, agentic AI usage within the research organization has continued to increase exponentially. At API list prices, the median researcher now uses around $2,400 USD per day on agentic AI tools, and the 90th percentile researcher now around $23,000 USD per day, up 62% and 105% respectively from $1,500 and $11,000 in September.2 This trend in increased agentic usage captures a wide variety of changes including increased activity, shifts to more expensive models, and changes to model behavior and agentic harnesses. — Read More

#strategy

Sashiko: Lessons from building an agentic code review system for Linux

The speed of development in large open-source projects has always been limited by code review, not code generation. Even as coding agents get better at writing code, they do not accelerate development on their own. This is especially true of patches to the Linux kernel. Without a way to build trust in proposed changes, more generated code simply piles up in front of the same small group of human maintainers. Some of those patches are solid changes, just written faster; many others are plausible-looking “AI slop” sent by contributors who do not fully understand the systems they are modifying. And the influx of AI-generated security reports, many of which are duplicates or inaccurate, only adds to maintainers’ load.

…To help maintainers handle the load and catch bugs before they land, the Google AI & Infrastructure team built Sashiko, an open-source, agentic code review system powered by Gemini. — Read More

#devops

Introducing Frostbite: A new CPU architecture discovered, designed, and verified end-to-end by AI

In our previous post, Redwood, we showed that given an architecture, our AI systems could implement, verify, and deploy an inference accelerator in record time. However, similar to other scientific domains, we believe that AI systems, given the correct freedom and guardrails, can co-design software and hardware together, making optimizations across the stack and opening a design space so large it was previously inaccessible. We set out to answer the question: “Could our AI systems & tools lead to a novel hardware discovery that radically outperforms human-made designs?”

Today, we’re introducing Frostbite, a general-purpose CPU with a new instruction set built from the ground up. Frostbite exploits the properties of interaction nets, namely that every step is local, every dependency is explicit, and every value has exactly one owner, to reach a step-function gain in CPU performance that has long stagnated. — Read More

#strategy

Why a Computer Scientist Thinks Superintelligence Won’t Happen

Many people were caught off guard this past summer when employees of one of the world’s leading AI companies began casually debating whether their products would lead to human extinction. It started when Anthropic researcher Jacob Coxon announced his resignation, warning that his employer and OpenAI were “racing straight to self-improving superintelligence and gambling with our lives.” Evan Hubinger, Anthropic’s head of alignment science, replied, “Jacob is correct here—we really do earnestly believe AI could kill all humans!” And Anthropic CEO Dario Amodei subsequently told CNN, “I agree with Jacob much more than I disagree with him.”

These concerns stem from a belief that has circulated in futurist circles for many decades: Once engineers design an AI system capable of improving itself, it will trigger an accelerating cycle of autonomous upgrades known as recursive self-improvement (RSI). The result will be artificial superintelligence (ASI), a system with cognitive capabilities so vastly exceeding those of humans that it will easily evade our control and perhaps even wipe out our species. Artificial superintelligence is, it should be noted, a far more ambitious goal than artificial general intelligence (AGI), which generally refers to AI that can match human capabilities across a broad range of cognitive tasks.

… I recently explored this topic in a computer science doctoral seminar that I taught at Georgetown University, in Washington, D.C., in which my students and I examined the technical and cultural roots of the superintelligence story. (For those who are interested, we published an annotated bibliography of our class readings.) I came away with several key objections to the feasibility of ASI, which I’ll detail below, as well as a deeper fear that even if the promised intelligence explosion never arrives, pursuing it can still create real harm. — Read More

#singularity

Who Builds the Next Model?

At Anthropic, Claude now writes more than 80% of the code the company merges. At Google, an AI agent called AlphaEvolve found a way to make Gemini train faster. At OpenAI, an “automated research intern” now carries out research tasks under human direction.

Put those together and a pattern appears. AI is starting to help build the next version of AI. Researchers call this recursive self-improvement. AI improves itself, the better version improves itself again, and the cycle repeats. — Read More

#training

Reflections on Trusting AI: A Virus Inside an LLM

Forty-one years ago Ken Thompson explained at the ACM Turing Award lecture how he had planted a self-reproducing backdoor in a C compiler that compiled a C compiler that ultimately compiled login.c. …This is a self-propagating backdoor that lives in the compiler.

… Poisoning a training set is not new. …What we show goes a step further: The poison does not sit in one dataset. It lives in a helper model and writes itself into the training data of
the next model, which becomes a helper and does it again. Self-propagating poison, not static poison! — Read More

#cyber

Introducing Devin Bots: from tasks to outcomes

A coding agent can finish every task you give it while still leaving you responsible for moving the project forward. Someone has to identify the next problem, investigate possible solutions, and check whether the changes actually helped.

We built Devin Bots to take on more of that responsibility. A Bot is a persistent agent with its own memory and tools that can lead multiple Devins in parallel. You can give it a large project or an ongoing goal. It investigates, delegates work, and evaluates the results to decide what to pursue next. It retains context across sessions and brings you only the decisions that need your judgment. — Read More

#devops

Nvidia is abandoning China after Jensen Huang’s ouster

Nvidia will no longer sell its H200 chips in China, acting CEO Colette Kress said Tuesday, ending the company’s participation in a controversial loophole in the U.S. ban on the sale of advanced chips to the country.

China’s access to leading AI accelerators was an important topic during the company’s first earnings call since CEO and co-founder Jensen Huang stepped down last week. The disclosure comes the same day Nvidia announced it was creating a board-level committee to oversee national security policies, export control compliance, and global trade risks. — Read More

#china-ai, #nvidia

AI Data Centers and the MIT Beer Game

Four people run a beer supply chain. Customer demand changes once, by four cases, and the whole thing still goes haywire. Now look at how AI data centers are being financed.

… The AI data center business is playing the same game but with more than four different actors. People and businesses buy AI products. Those products run on models from labs like OpenAI and Anthropic. The labs rent compute from the hyperscalers and from the neoclouds – CoreWeave, Nebius, and a bunch of others whose whole business is renting out GPUs. The clouds buy chips from Nvidia, which buys wafers, memory, and packaging from its suppliers. Each actor sees orders from the actor below it, but none of them see actual end demand. Even if they think they do, they are only guessing at some future state. And, in this game, the delays aren’t measured in weeks. Land, power, permits, and construction take years. Elections and geopolitics change regulatory dynamics, which introduce other delays. This week, the Commerce Department closed a loophole that let Chinese companies rent Nvidia computing power located in other countries. The cost of capital moves with the economy, the risk, and, more importantly, how people perceive future risk. — Read More

#strategy