Articles tagged "SWE-Bench"

**Anthropic released Claude Sonnet 5 on June 30, 2026** ([official release](https://www.anthropic.com/news/claude-sonnet-5) / [System Card](https://www.anthropic.com/claude-sonnet-5-system-card) / [TechCrunch](https://techcrunch.com/2026/06/30/anthropic-launches-claude-sonnet-5-as-a-cheaper-way-to-run-agents/) / [VentureBeat](https://venturebeat.com/technology/anthropic-launches-claude-sonnet-5-at-a-steep-discount-to-its-top-model-as-the-company-races-toward-a-blockbuster-ipo)). **The headline: the mid-tier Sonnet just leapfrogged Opus 4.6 by 12 points** — **92.4% on SWE-Bench Verified** (Opus 4.6 was 80.8%), **88.3% on OSWorld-Verified** (15.9 pts ahead of the 72.4% human-expert baseline), **96.2% on GPQA Diamond** (over [Gemini 3.1 Pro's](../columns/local-llm-landscape-2026-june-update) 94.3%), and **84.7% on ARC-AGI-2** (7.6 pts ahead of Gemini 3.1 Pro's 77.1%). It ships with a **1M-token context window** (matching Opus 4.8) and a 128K max output. **Strategic pricing on the eve of Anthropic's IPO**: **introductory $2 / M input and $10 / M output through August 31, 2026**, after which standard pricing becomes **$3 / $15** (matching [Sonnet 4.6](../columns/claude-agent-sdk-credit-billing-change-2026-06-15)). Note: **a new tokenizer maps the same input to about 1.0–1.35× more tokens**. It undercuts GPT-5.5, Gemini 3.1 Pro, and Anthropic's own Opus 4.8 on price. **Default-model rollout**: now the **default in claude.ai Free and Pro**, **default in Claude Code Pro**, and available via API (`claude-sonnet-5`), AWS Bedrock, Vertex AI, and Managed Agents. Zapier's Daniel Shepard told TechCrunch that **"earlier Sonnet versions would stall on multi-step tasks — Sonnet 5 finishes them end-to-end."** **Safety**: lower misalignment than Sonnet 4.6, cyber safeguards on by default, and a **0.0% exploit-creation rate** on Firefox vulnerability tests. **Strategic context**: agentic capability is now "table stakes" across foundation-model companies; competition has shifted to **cost-efficiency, reliability, and autonomous-task completion**. Heading into its IPO, Anthropic is breaking the boundary between its Opus and Sonnet tiers to **win the cost / performance contest in high-volume production workloads**.

**DeepReinforce released Ornith-1.0 on June 26, 2026** ([official](https://deep-reinforce.com/ornith_1_0.html) / [Hugging Face collection](https://huggingface.co/collections/deepreinforce-ai/ornith-10)). It is an **MIT-licensed open-weights family specialized for agentic coding**, **with no regional restrictions**. **Three sizes**: [Ornith-1.0-9B](https://huggingface.co/deepreinforce-ai/Ornith-1.0-9B) (dense, ~19GB BF16) / [Ornith-1.0-35B](https://huggingface.co/deepreinforce-ai/Ornith-1.0-35B) (MoE) / [Ornith-1.0-397B](https://huggingface.co/deepreinforce-ai/Ornith-1.0-397B) (MoE, built on Qwen 3.5 + Gemma 4). **All sizes ship 262K context**, with **FP8 and GGUF quantizations released alongside**. **Benchmarks (vendor-reported, claimed SOTA at each open-weights size tier)**: | Benchmark | 9B | 35B | 397B | |---|---|---|---| | **SWE-Bench Verified** | **69.4%** | **75.6%** | **82.4%** | | **SWE-Bench Pro** | **42.9%** | **50.4%** | **62.2%** | | **SWE-Bench Multilingual** | — | — | **78.9%** | | **Terminal-Bench 2.1** | 43.1% | 64.2% | **77.5-78.2%** | | **NL2Repo** | 27.2% | 34.6% | **48.2%** | | **ClawEval** | — | — | **77.1%** | **Design thesis**: Reinforcement learning optimizes **both the solution rollouts and the scaffolding (the agent structure that drives them) itself** — a 'self-improving' agentic-coding design. It sits naturally next to the [Loop Engineering Maker-Checker](../columns/loop-engineering-ai-agent-paradigm-2026-06) paradigm. Reasoning is exposed via `<think>...</think>` blocks; function calling and tool use are first-class. **Distribution and ops**: vLLM ≥ 0.19.1 / SGLang ≥ 0.5.9 / Transformers ≥ 5.8.1 / Docker + llama.cpp / Ollama. OpenAI-compatible endpoints. The 9B fits on a single 80GB GPU; 35B and 397B want an **8×80GB GPU node (TP=8)**. Agent-framework compatibility: **OpenHands, Hermes Agent, and [OpenClaw](../services/openclaw-setup)** (Oflight's own service line — and ClawEval is in DeepReinforce's published benchmark set). **DeepReinforce lineage**: an RL-focused research organization that has previously shipped [CUDA-L1 (avg 3.12× GPU speedup)](https://github.com/deepreinforce-ai/CUDA-L1), [CUDA-L2 (HGEMM kernels beating cuBLAS)](https://github.com/deepreinforce-ai/CUDA-L2), and **IterX (MLSys 2026 NVIDIA Track)**. Ornith-1.0 applies the same RL playbook to LLM self-improvement. **Positioning**: alongside [Kimi K2.7-Code](../columns/kimi-k2-7-code-moonshot-ai-2026-06) (1T MoE / 32B active) and [GLM-5.2](../columns/local-llm-landscape-2026-june-update) (Intelligence Index v4.1 = 51, open-weights leader), **Ornith-1.0 is at the front of the June-2026 agentic-coding open-weights race**. Against Chinese-origin models (Kimi / GLM), its differentiator is **MIT license + no regional restrictions + a US-flag procurement story**. **Caveat**: benchmarks are DeepReinforce's own vendor-reported numbers. Independent third-party verification on public leaderboards has not yet appeared (as of June 26, 2026). The article closes with **three inquiry funnels for Ornith-1.0–era local-LLM evaluation, build, and ongoing maintenance**.