I love tests. I have always liked tests, and AI has made me think about testing even more. If AI is going to help us write production code, the important question is not whether it can generate a lot of code quickly. The important question is whether we can build a feedback loop that tells us what is actually working.

That was the reason for my talk, “When Tests Lie: Using Observability to Keep AI Honest.” I wanted to understand what happens when you use AI on something much larger than a toy project.

That was part of the experiment. I wanted high quality code. I had architectural opinions. I cared about security, performance, and distributed systems behavior. I also wanted to stay human in the loop, because I wanted to know what went wrong rather than automate everything away from the start. — Read More

Sierra’s Mihai Parparita published field notes on building the company’s internal MCP gateway, and the framing is the most honest thing in it: MCP is “ancient” technology by AI-industry standards, so wiring agents to internal systems should have been straightforward, and instead it became “another engineering iceberg”. The tip above the waterline is the part that demos well: 89% of Sierra employees now connect agents to 45 different services from a single page. The mass underneath is auth.

It’s the best field report on internal agent platforms this year, and every team standing up an agent gateway should read it before writing a line of code. But walk the seven lessons in order and a pattern shows up. The expensive ones are barely about MCP at all. They’re about identity, scopes, consent, and audit. Enterprise software has worked on those four for twenty years, and agents have made all four urgent again at once, in a context where the old answers do not transfer cleanly. — Read More

Tell us the job.We’ll find the right model. Neurometric now has a new site where you can compare the price of any task on various models. — Read More

#devops

It’s a mess, right? If you have not been thoroughly disappointed in AI capabilities, you have not tried enough. And the expectations grow. We’ve been through the ladder of prompt engineering, context engineering, harness engineering. Then Steinberg is tweeting about Loops one month, Graphs the next.

Should your software factory be more autonomous, more dark? Have you sacrificed quality for speed under pressure to deliver? Or is your job now dealing with what happens when other people trade quality for speed? Is AI output mostly just rubbish? Is it all a huge mess? Yes, maybe, maybe not. — Read More

Your AI agent just finished a coding task.

… A few hours later, you notice something strange.

… We keep trying to make AI agents smarter.

But what if the model isn’t the biggest problem?

What if the real problem is the environment we’re putting it in?

That’s where agent harnesses come in. — Read More

We previously noted that, while it’s easier than ever to hit a particular quality bar by having coding agents use effective test techniques, software quality seems to be getting worse, indicating that whatever defaults developers are using may not work very well. Here, we test if simple instructions to agents to use particular techniques or libraries improve implementation correctness, as a kind of test to see how effective agents are when guided by someone with no expertise in testing who’s maybe heard that you should apply certain techniques or use certain libraries. — Read More

#devops

I used to buy books to learn something. I would read, highlight, take notes, and file them somewhere I meant to come back to. Then weeks later, in the middle of a real project, I would try to remember what the book said.

Most of it never made it across that gap. The reading and the applying sat too far apart.

… There is a shorter path now. You can turn a book into a skill your agent loads while you work, so the framework arrives at the moment you need it. Convert it once and the book stops being something you try to remember and becomes something that runs. — Read More

Output quality is important when working with AI coding agents, but true efficiency comes from getting work done quickly, efficiently, and with the right context.

That’s why token count of individual interactions alone isn’t a meaningful measure of efficiency. The goal shouldn’t be to use fewer tokens, but to tap into the right amount of context to move a task forward. — Read More

A language model can perform well on a clean benchmark and still struggle with the cases that matter in production.

But as a system moves closer to production, the evaluation problem changes.

Real inputs are often ambiguous. Labels may be inconsistent. Important context may be missing or truncated.

We encountered these challenges while evaluating an LLM-based system designed to reduce false positives in GitHub secret scanning.

Rather than determine whether an LLM could classify a string correctly, we needed to understand whether the system could reduce noisy alerts while preserving enough recall to remain safe for a security workflow. — Read More

The traditional software development lifecycle (SDLC) is process-heavy to ensure accountability and control at each step. However, the traditional SDLC was designed to maximize efficiency in an era where the most time-consuming and expensive stage was writing and implementing code, which is no longer the case. PRDs, estimation rituals, and product security reviews all existed to force alignment during what could be weeks, months, or quarters of development work.

To better realize the productivity gains of and secure agentic AI, the traditional SDLC lifecycle requires the same level of transformation as the implementation phase has undergone. — Read More