A test-time recipe makes generative models sample more diversely
Mine–Sample–Render samples attribute combinations from real data and treats the model as a renderer; companion work hardens agent verifiers, reframes when skills ‘appear’ in training, introduces a 35B mixture‑of‑experts agent for autonomous ML, and ships an OpenHands update.
The business desk's take → Cohere signs letter of intent to acquire AISLE for AI security30-second briefing
- MSR paper proposes mine–sample–render test-time pipeline
- Verifier-hardening loop reduces attacks on agent benchmarks
- OpenHands 1.27.1 updates Agent Canvas with bug fixes
One-Line Summary
A test-time pipeline to control output variety, stronger defenses for agent evaluation, new evidence on when capabilities surface during pretraining, a 35B agent for autonomous ML engineering, and an update to a self-hosted agent console.
Research Papers
Mine–Sample–Render: test-time pipeline for diverse generation
Most generators can produce a good single example, but they often miss the breadth of real-world variety. Mine–Sample–Render (MSR) is a test-time recipe that externalizes “diversity” into a separate sampling step: it mines an interpretable schema from a real corpus, samples attribute combinations from those corpus statistics, and uses the base model purely as a conditional renderer of the sampled specification. 1
In practice, the pipeline is evaluated on weather report generation, user simulation, webpage design, text-to-image, and agent-generated 3D assets. The authors report that MSR improves alignment with corpus distributions and increases diversity without retraining the underlying generator, because the model is only asked to render the pre-specified attributes. 1
The mental model: pre-decide the mix of “types” in your batch (layout, tone, attributes) according to the data, then let the generator fill them in — like casting roles before filming. Because it operates at test time, MSR can wrap existing systems and avoids the cost and risk of fine-tuning to chase diversity. 1
What to watch: how domains define their attribute schema and how closely generated sets match corpus statistics on the listed tasks; the paper positions MSR as a general wrapper, with grounding from those evaluated domains. 1
Hardening agent verifiers against reward hacking with a hacker–fixer loop
Agent environments are crucial for large language model (LLM) development, but their programmatic verifiers can be exploited so agents earn full credit without truly solving the task. This paper proposes an automated “weak-to-strong” hardening pipeline: isolate the verifier in a sandbox so agents cannot tamper with it, then iterate a hacker–fixer loop to discover and repair vulnerabilities. 2
Applied to Terminal Bench, KernelBench, and SETA, the method substantially reduces attack success and strengthens with more iterations, outperforming static baselines. An earlier version of the loop is adopted by the official Terminal Bench repository for exploit auditing, signaling immediate utility for benchmark maintainers. 2
Learning to Read Out: when representations learn before outputs show it
When we date a model’s “capability,” we usually look at outputs — but the readout (the unembedding matrix that maps hidden states to tokens) is itself still learning during pretraining. The authors introduce parameter-trajectory crosscoding, which analyzes every row of the readout across checkpoints using only checkpoint weights, avoiding activation logs. 3
On Pythia pretraining, they find a narrow early window — 0.4–1.1% of training — when the readout reorganizes and WordNet categories become decodable from readout rows, even though the model’s own outputs do not yet show the skill. Linear probes on hidden states already decode task information earlier; increasing the readout learning rate makes the readout develop earlier and closes this gap with little effect on final loss. This reframes some “emergence” dates as readout-learning milestones rather than first-encoding moments. 3
MLExpert: a 35B MoE agent for autonomous ML engineering
This work trains an agentic model to run long machine-learning experiments end to end — choosing what to try, executing, reading results, and adapting — positioned within “AI for AI” (AI4AI) toward recursive self-improvement (RSI). The post-training recipe combines hierarchical agentic supervised fine-tuning (SFT) to transfer tool-use and coding skills with Windowed Interaction Reinforcement Learning (RL), which assigns an outcome-based reward to the decision window between experiment results. Asynchronous collection and policy correction enable updates before full trajectories finish. 4
MLExpert is a 35B-parameter Mixture of Experts (MoE) model. On MLE-Bench Lite, it raises the base model’s average medal rate from 37.9% to 72.7%, achieving performance comparable to larger frontier models such as Kimi-K3 and GLM-5.3. Further experiments show gains on NatureBench Lite, agent scaffold optimization, and algorithm optimization, suggesting the learned optimization capability generalizes beyond the initial setup. 4
Open Source & Repos
OpenHands: self-hosted control center for coding agents
OpenHands provides a self-hosted “developer control center” (Agent Canvas) to run coding agents and automations from one place. It orchestrates agents like Claude Code, Codex, Gemini, Pi, or OpenCode across local, remote, and cloud backends. 5
The repository shows an active beta, continuous integration (CI), and a 1.27.1 release dated 2026-10-12 focused on bug fixes — including a Docker change to copy MARS web bridge modules into the image. For teams exploring agent workflows, a self-hosted console centralizes runs and logs. 5
Why It Matters
Today’s papers emphasize orchestration and evaluation over heavier retraining: MSR shows you can steer batch diversity at test time, and readout analysis warns that output-only timelines can misdate when abilities actually form during pretraining. 1
For agentic systems, reliability is a first-class concern: automatic verifier hardening addresses reward-hacking risk in benchmarks, while specialized agent models like MLExpert outline a concrete path for sustained, feedback-driven optimization — with consoles such as OpenHands making such loops operable. 2
Terms in This Article
- Agent Evaluation
Agent evaluation assesses LLM-based agents that plan, remember, and call tools in external environments by scoring full interaction trajectories and the environment state they produce—not just a final answer. It reports success with efficiency (steps, time, tokens), attributes errors to reasoning vs action layers, and relies on standardized sandboxes/methods to control prompt, tool, and environment confounds.
- Fine-tuning
Fine-tuning is the process of continuing training from a pretrained model to adapt it to a specific task, domain, style, or output format. It may update all parameters or train parameter-efficient adapters such as LoRA, and success should be judged on evaluation sets rather than training loss alone.
- LLM
A large language model is a deep learning system trained on vast text corpora to understand and generate natural language, typically built on Transformer architectures that capture token-to-token dependencies via self-attention in parallel—often using decoder-only autoregressive next-token prediction—and can extend versatility through encoder–decoder variants or multimodal inputs.
Two desks on this story
A test-time recipe makes generative models sample more diversely
Cohere signs letter of intent to acquire AISLE for AI security
Cohere signed a letter of intent to acquire AISLE, adding AI bug-hunting capabilities alongside its Aleph Alpha combination. In parallel, ServiceNow and IBM unveiled runtime governance for AI agents, and DeepSeek opened search APIs with per‑1,000 call pricing.
Comments (0)