New benchmark tests how coding agents find the right files
Agent Retrieval Bench measures repository context retrieval across 25 codebases with 427 samples and finds no single method consistently wins. Also in focus: CodeNib’s multi-view context system, Google’s ADK 2.0 toolkit, and Rapid-MLX speed claims on Apple Silicon.
One-Line Summary
Coding agents are being graded on how well they find the right files before they code, while new systems and toolkits aim to make that context cheaper and faster to serve.
Research Papers
Agent Retrieval Bench measures how coding agents find files
Before a coding agent writes a fix, it has to pull the right files from a large codebase — Agent Retrieval Bench is a testbed that evaluates this upstream step. It freezes repositories at specific base commits and scores retrieval by what an agent needs next, not just by text similarity between a query and a file. The corpus spans 25 repositories with 427 samples, including 308 base-commit snapshots, 392,000 files, and 7.9 million chunks. 1
The benchmark covers four positive-retrieval tasks — code2test, comment2context, trace2code, edit2ripple — and adds a selective-retrieval subset with natural “no-gold” cases and counterfactual wrong-repository controls. Of the 427 samples, 345 are positive examples, 50 are natural no-gold, and 32 are counterfactual controls. 1
Key result: no single retrieval family dominates. Qwen3-Embedding-4B leads on sample-weighted MRR (a ranking-quality score) for positive samples, Qwen3-Embedding-8B has the best Recall@20, and RepoMap yields the most useful context under an 8K-token budget — with winners shifting by task. In plain terms: the best method depends on the job and the context window you can afford. 1
Two failure modes stand out for teams shipping agents. First, selective thresholds calibrated on counterfactual controls do not improve selective success on natural no-gold cases, exposing a calibration gap. Second, logged agent trajectories miss every gold file on 27–35% of samples. A seed-intervention pilot shows retrieval-derived initial context boosts file F1 and reduces exploration versus random non-gold seeds, while “oracle” gold seeds show substantial headroom remains. 1
Latent strategies: separating a model’s reasoning styles
This paper tries to tease apart the different ways a Large Language Model (LLM) solves a problem by learning a hidden “router” that picks a strategy before the model writes its answer. Formally, it factors the base response distribution into a router over latent strategies and a generator conditioned on that strategy. 2
A core challenge is posterior collapse: the generator, initialized from the base model, can ignore the strategy variable and still match the training loss. The authors propose a variational objective that measures fractional information gain relative to the base model’s loss and focuses reconstruction on high‑surprisal tokens, encouraging the latent code to carry strategy-relevant variation. 2
On a benchmark of multi‑strategy algorithmic tasks, the method recovers latent codes aligned with distinct reference strategies while preserving the base model’s overall response distribution. For practitioners, this points to more controllable routing without sacrificing the answers users already expect. 2
CodeNib consolidates repo search and context serving
Coding agents repeatedly search, navigate, and keep context as repositories evolve; CodeNib packages this into one runtime by precomputing lexical, dense, and structural views per commit and serving ranked search, symbol navigation, and bounded context. Across 100 snapshots, when outputs match an independent rebuild, graph and vector updates are 8.7× and 25.4× faster at the median, mapping quality‑cost frontiers across the repository‑context lifecycle. 3
On static‑navigation requests that match normalized live‑server locations (63% of 1,000 requests), the median per‑request live/static latency ratio is 4.7×, indicating substantial speedups from prebuilt views. Across five models, selected‑context policies preserve localization with 50–87% fewer trajectory tokens than paired grep/read, meaning you can send less text to the model and still stay on target. 3
The system sets explicit, operation‑specific validity boundaries for each view, clarifying when to trust static indexes versus live sources. That framing makes costs and correctness visible — useful for teams wiring agents into IDE‑like experiences. 3
Open Source & Repos
Google’s ADK 2.0 ships a code-first agent toolkit
Google’s Agent Development Kit (ADK) 2.0 is an open‑source, code‑first Python framework for building, evaluating, and deploying sophisticated AI agents with more flexibility and control than GUI‑heavy tools. The repository highlights Apache‑2.0 licensing, a PyPI package, and maintained unit tests, with documentation linked from the README. 4
If you are standardizing agent scaffolding across projects, ADK’s focus on evaluation and deployment primitives can reduce glue code and improve reproducibility from prototypes to production. The badges and packaging signal an effort to meet developers where they already work — Python, pip, and CI. 4
Rapid-MLX promises faster local models on Apple Silicon
Rapid‑MLX is a local AI engine for Apple Silicon Macs that the project claims runs 4.2× faster than Ollama and achieves a 0.08‑second cached Time To First Token (TTFT) — the delay before the first generated token appears. It advertises 100% tool calling, 17 tool parsers, prompt caching, “reasoning separation,” and a drop‑in Application Programming Interface (API) replacement that works with tools like Claude Code, Cursor, and Aider. 5
Distribution badges point to both PyPI and Homebrew, and the latest tagged release is v0.11.1 (2026-07-28) with changes like restoring benchmark baselines and improving first‑run behavior. Treat the speed claims as project‑reported until you validate on your hardware and prompts. 5
Community Pulse
Hacker News (198↑) — Commenters highlight cost barriers to iterative agent workflows and skepticism about industry‑funded research, alongside interest in cheaper, tailored models to reduce dependence on closed services. 6
"This is a bigger issue than folks realize, visual inputs to GPT4 are really expensive (like several cents per dozen images in some cases), which means that you can't just spam the API to iterate on HTML/webpages with a software agent. We're trying to tackle this for web screenshots (also documents) with a custom model geared towards structured schemas designed to be fed into a feedback loop like the above while keeping costs down." — Hacker News 6
"It’d be really nice to see research in this area from somewhere without a financial interest in hyping AI. That incentive doesn’t invalidate research, but AI results are so easy to nudge in any direction that it’s hard to ignore." — Hacker News 6
Why It Matters
Agent performance often hinges on context acquisition — not just code generation. Today’s results quantify that gap (no single retriever wins; calibration is tricky) and offer concrete systems approaches (multi‑view indexes, bounded context) that make retrieval faster and more predictable in real repositories. 1
Tooling is maturing in parallel: Google’s ADK 2.0 formalizes agent development workflows, and local engines like Rapid‑MLX aim to cut latency and cloud costs for everyday use — a pragmatic counterweight to purely cloud‑based stacks and an on‑ramp for faster iteration. 4
Comments (0)