Recent progress in AI has made software engineers dramatically more productive. However, our experience building and deploying Devin suggests that agentic software engineering remains far from solved. Models struggle to infer user intent, respect unstated constraints, and anticipate long-term consequences of their changes, forcing human intervention.

Most existing coding benchmarks fail to capture these gaps and tend to be saturated quickly. We are excited to introduce FrontierCode 2.0, a new generation of our coding benchmark series and our best attempt yet to evaluate user experience. — Read More