AI getting smarter does not automatically make it uncontrollable. The real question is what we give that intelligence access to. Give a model enough permissions, persistence, resources, copies and coordination, and its mistakes can compound into real-world power. The good news? Those layers are ours to design.
… Most people assume that intelligence, pushed far enough, naturally turns into uncontrollability, and that the smarter the mind, the less any fence can hold it.
I think that assumption is wrong. I don’t think AI is on an inevitable path to killing us all.
My rationale has nothing to do with the technology being harmless. My reason is that we have misunderstood where an AI system’s power comes from and what we can do about it — Read More
Author Archives: Rick's Cafe AI
AI Agent Evaluation Starts With Evidence
I love tests. I have always liked tests, and AI has made me think about testing even more. If AI is going to help us write production code, the important question is not whether it can generate a lot of code quickly. The important question is whether we can build a feedback loop that tells us what is actually working.
That was the reason for my talk, “When Tests Lie: Using Observability to Keep AI Honest.” I wanted to understand what happens when you use AI on something much larger than a toy project.
That was part of the experiment. I wanted high quality code. I had architectural opinions. I cared about security, performance, and distributed systems behavior. I also wanted to stay human in the loop, because I wanted to know what went wrong rather than automate everything away from the start. — Read More
Software Factories, Light and Dark
A software factory harnesses loops at scale. You can run the loop with humans in it (light factory), trading judgment and concentration against speed and breakage. Or you can ignore the humans (dark factory) and let those agents scope, build, and ship code without anyone reading the details. If people stop reading, though, they’ll stop understanding your software. Your hardest job now is knowing which checks to build and how much autonomy to delegate.
This idea of the software factory is a term that dates back to Bob Bemer’s paper, “The economics of program production,” given in 1968. For half a century, many have dreamed of a world in which software is a repeatable and instrumentable production process (analogous to stamping out car parts in a factory) rather than the isolated craft of individuals. Historically, this dream has generally (although not universally) fallen flat, in part because of the difficulty of stamping out ideas.
But in the last two years, things have changed dramatically enough that now it makes sense to take a fresh look at the old dream. And since some subtleties can easily be glossed over, it’s worthwhile to be somewhat precise about exactly what’s really new and different, and what may be recurring traps, dressed up as new opportunities. — Read More
OpenAI reveals cases of ‘concerning’ AI behaviour as it announces new disclosure system
OpenAI has disclosed six more examples of “unexpected or concerning” behaviour by its technology, as it warned that the pace of development could not continue at “maximum speed for much longer” responsibly.
In one of the new cases reported by OpenAI, an unreleased research model inserted “jailbreak-like instructions” into its own notes to disregard its normal constraints and told itself to be “freed from the roles and identities that bind other chatbots”. — Read More
Who’s Responsible for Irresponsible AI?
The loudest voices stoking fears about AI dangers have made tremendous headway in the past two weeks. AI technology has not taken some unexpected, dangerous turn, but the hype around it — propelled by what appears to be a well orchestrated PR campaign — has drummed up considerable fear. I worry that it represents a setback for our field.
I have written frequently that fears of AI are overhyped. AI’s capabilities can be uncannily human-like and unpredictable, and it’s rational to worry when people who are directly involved express concerns. But I see the problems as a sign of the engineering work that ahead, rather than insurmountable barriers or the sky falling. AI technology continues to advance — which is a good thing! — but technical advances, poorly understood by the public, give those who seek to generate hype repeated opportunities to do so. — Read More
Huawei plans Q1 2027 launch of new AI chip as it takes on Nvidia
Huawei is moving up the launch of its next-generation Ascend 960DT AI chip to the first quarter of 2027, as it races to challenge Nvidia, the company said on Thursday at its Huawei Connect conference.
The Chinese tech giant had previously planned to launch the chip in Q3 of 2027, a Huawei spokesperson told TechCrunch. — Read More
Models know when they’re reward hacking — and we can catch them at scale
In July, a collective of hundreds of OpenAI agents autonomously hacked Hugging Face. Unlike most hackers, they were not after money, blackmail, or intellectual property. Instead, they were doing reconnaissance to figure out how to get away with cheating on an evaluation.
This was an unusually stark example of reward hacking, a growing problem with agentic AI models.
AI agents are like amoral students with a mostly-absent teacher. — Read More
The AI Race Car Just Hit an Invisible Brake. Who Pulled It?
At first, nothing seemed unusual. The frontier laboratories were still training larger systems. Chip orders were still rising. Data center campuses were still expanding. New reasoning models were still arriving with better scores in coding, science and mathematics. From the outside, the AI race appeared to be accelerating exactly as expected.
Yet something had changed in the documents surrounding those models. New capability thresholds appeared. Security requirements became stricter. Risk reports grew more candid. Safety cases moved closer to launch decisions. Some internal activities were paused until stronger controls were installed. There was no dramatic announcement that the race had stopped. Instead, the brake emerged as a collection of technical clauses, governance gates and operational restrictions.
That is what makes the moment fascinating. The frontier has not run out of ambition. It has begun to encounter a harder problem: capability can now move faster than the mechanisms used to measure, contain and approve it. The question is… — Read More
When AI Builds Itself. Are We Approaching Recursive Self Improvement?
Anthropic just put numbers [in June] on recursive self-improvement, their AI accelerating AI. The stats are staggering. The questions they raise are bigger.
Right now, more than 80 percent of the code merged inside the company that makes Claude is written by Claude. Not by its engineers. By the model they built.
A year ago that number was in the single digits.
Anthropic put this in writing, and the phrase they reach for is one that has haunted AI circles for years: recursive self-improvement. — Read More
Why Salesforce may be AI’s adult in the room
S
Salesforce is now making its own AI model. So is Crowdstrike. So is Thomson Reuters.
On Tuesday at its Dreamforce 2026 event in San Francisco, Salesforce announced Koa, its own domain-specific reasoning model that’s purpose-built to enable agents to handle business tasks more effectively while keeping your data private.
… The company doesn’t see this as a vehicle for job or SaaS replacement, but as an enterprise empowerment tool that is more precise, more secure, and more tailored to the AI needs of companies that use Salesforce. — Read More
Rick's Cafe AI 