A language model can perform well on a clean benchmark and still struggle with the cases that matter in production.

But as a system moves closer to production, the evaluation problem changes.

Real inputs are often ambiguous. Labels may be inconsistent. Important context may be missing or truncated.

We encountered these challenges while evaluating an LLM-based system designed to reduce false positives in GitHub secret scanning.

Rather than determine whether an LLM could classify a string correctly, we needed to understand whether the system could reduce noisy alerts while preserving enough recall to remain safe for a security workflow. — Read More

The traditional software development lifecycle (SDLC) is process-heavy to ensure accountability and control at each step. However, the traditional SDLC was designed to maximize efficiency in an era where the most time-consuming and expensive stage was writing and implementing code, which is no longer the case. PRDs, estimation rituals, and product security reviews all existed to force alignment during what could be weeks, months, or quarters of development work.

To better realize the productivity gains of and secure agentic AI, the traditional SDLC lifecycle requires the same level of transformation as the implementation phase has undergone. — Read More

Slack wants to drag AI coding out of the terminal and into the group chat.

The Salesforce-owned messaging platform today announced Slack Code, a new product that embeds AI coding agents — including Anthropic’s Claude Code, Cognition’s Devin, GitHub Copilot, and Vercel’s agent — directly into dedicated Slack channels where entire teams can watch, steer, review, and ship software together. Slack Code is available on any Slack plan at launch, though customers need their own access to the partner agents.

The pitch is deceptively simple: today, most work with AI coding agents happens between one person and one agent, invisible to everyone else. Slack Code makes that work “multiplayer.” When someone tags a coding agent from any conversation, the agent spins up a project-specific code channel, does the work in the open — complete with code diffs, live previews, and a running plan visible in dedicated tabs — and archives the channel when the job is done, leaving behind a searchable audit trail. — Read More

In late 2025, we ran an experiment to answer one question: “Can coding agents autonomously solve a production-grade problem from scratch on their own?”

For this, we tasked two agents with the (at that time) best publicly available coding models with a real problem and a real deadline. The result of this experiment is a tokenizer trainer called toktoktok, and is now open source on GitHub.

In this article, we share what we learned about designing effective loops that allow agents to autonomously solve production-grade problems: how to specify a goal for multi-domain experts and how to set up the verification infrastructure. — Read More

The manifestation of my imposter syndrome, for me and today, is what does it mean to be a software engineer. There’s a lot more noise than signal on the Internet about agentic engineering, what can be accomplished, and its implications for the future. The title I chose rather gives it away; it’s about choosing — carefully — all the things you need to choose when you’re solving the puzzles of software and systems development.

… In the past year, agent harnesses crossed the “can it be done” rubicon. … The economic models for the large models aren’t viable from any report that I’ve seen, but the capability isn’t going away. Instead it’s shrinking (fast!). Open weight models are making (beefy) personal computers quite capable of doing the same. They’re not quite as effective, but the delta in time and capability isn’t large. — Read More

The UI is dead. Or so I keep hearing:

“Agents are your users now, software is losing its head, and everyone who learned Figma should start learning Bash.”

… Here’s my take:

— Headless is a new layer on top of product building, the same way APIs and the GUI once were
— Everything is a user interface if I’m using it
— My agents are using it too now, just not the part you designed

— Read More

“Done” is rarely a property of the work itself. It is a judgment produced by the system around the work. Humans do not possess a universal detector for “done”. We rely on a patchwork of signals like tests, specifications, precedent, approval, deadlines, risk, and finding that point of diminishing returns. In each case, completion comes from outside the work itself.

An AI model can almost always produce another answer.

Loop engineering is not the practice of making an agent retry. It is the practice of making each cycle reduce the distance between the current state and a desired state. A loop is not yet a direction. — Read More

Open-source coding models caught the frontier in 2026. DeepSeek V4 Pro recently hit a reported 80.6% on the SWE-bench Verified benchmark, sitting right next to the most expensive closed source models in the world.

But here is the catch. The single best open coding model right now needs a server rack you do not have. … The question is which model is best that you can run for the kind of coding you actually do.

Here are the 10 worth knowing. One will fit your machine. A different one will fit your wallet. — Read More

For the past decade, Azure DevOps has powered software development at Microsoft, supporting some of our largest repositories and most complex engineering workflows across Azure Repos, Boards, and Pipelines.

Software development is being reshaped by AI, and where code lives now have a direct impact on how much value organizations can capture. For teams that want to take full advantage of AI-native development, repository location is becoming a strategic decision.

Azure DevOps and GitHub product teams have spent the past few years building the integration, migration, and enterprise-readiness capabilities needed to give organizations on Azure Repos a path to unlock the full value of GitHub’s latest agentic capabilities.  — Read More

This morning I published The Framebuffer People, about cloning every OS posted to r/osdev and finding a pile of QEMU-shaped Linux reskins with Patreon buttons. The thread had two kinds of replies. The loud kind called me a gatekeeper. The quiet kind asked a real question: “Another way to look at it. Give me a suggestion. What do I build?”

In the thread I answered that I can’t answer, and that is half true. Nobody can tell you which part of an OS will grab you. But there is a path that produces understanding and a path that produces a README with a roadmap hallucinated by a model. Here is the first path, written down properly. — Read More