One-Line Summary

Today's papers shift attention from bigger models to smarter pipelines: design data mixtures like experiments, measure memory without lexical crutches, relay specialist solvers for code, and recycle failures into training signal.

Research Papers

Data mixing for large-model pretraining becomes an experiment by design

Choosing how much web, code, books, and other sources to feed into a large language model (LLM) is reframed as a classical “mixture experiment,” analyzed with response surface methodology (RSM). The authors fit sparse second-order Scheffé models to validation loss over the mixture simplex and propose model-robust I-optimal designs that tell you which small proxy trainings to run before scaling up. ¹

Applied to a RegMix case study, the analysis shows that domain value is relational: several domains that look weak in isolation become favorable through pairwise interactions, especially when combined with web-derived text. The sparse Scheffé model preserves mixture rankings across model scales and remains competitive with a flexible machine-learning predictor while explicitly separating additive from interaction effects. ¹

In a simulation calibrated to observed proxy-training responses, I-optimal designs recover the relevant mixture ordering after removing about 25% of the original proxy runs, indicating a meaningful gain in statistical efficiency. The paper argues data mixing should be treated not only as a prediction task but also as an experimental-design problem where the proxy mixtures themselves are the design points. ¹

For practitioners, this suggests running fewer but better-chosen proxy trainings to map the response surface, with clearer diagnostics for interactions between data sources. Watch for teams to adopt RSM in pretraining pipelines and to report mixture effects rather than only single-domain ablations. ¹

Tablet-2 evaluates non-lexical memory retrieval across languages and images

Tablet-2 is a production long-term memory engine evaluated on text benchmarks and on cross-lingual retrieval of captionless photographs, using no lexical matching, no keyword scoring, and no language model. It reports 95.7% [93.4, 97.1] on LongMemEval-S (500 questions) and 67.5% [64.8, 70.2] on BEAM-1M (700 questions, 2.21M stored memories), and shows that changing only the reader moves LongMemEval-S by 2.0 points while changing only the re-ask budget moves BEAM-1M by 8.9 — so the authors treat their tables as placement, not rankings. ²

For multimodal retrieval, Tablet-2 reaches 95.2% mean recall@5 over 70 store-and-query language cells where a strong lexical baseline, Okapi BM25 (BM25), averages 19.0% and yields exactly zero in 54 cells, since captionless photographs provide no text to score. Open dense baselines on Crossmodal-3600 show large language variance (e.g., 91.0% on English vs 4.7% on Russian from identical image vectors), while Tablet-2’s spread is 14.0 compared with baselines at 27.5 and 27.7; three negatives are also reported at equal weight: low-resource languages degrade (Swahili 53.0%, Telugu 64.0%), adding captions lowers cross-lingual retrieval by 11.4 points, and an omitted setting cost 37 points of Korean top-1 accuracy. ²

Takeaway: non-lexical memory can work across languages and modalities, but headline scores depend on evaluation knobs like the reader and retry budget. When comparing systems, hold engine, corpus, and judging constant — or report the sensitivity explicitly. ²

MARS relays specialist models for competitive programming

MARS is a prompt-only pipeline that tackles competitive programming by relaying work among topic specialists — dynamic programming, graphs, strings, geometry — each powered by a large language model (LLM) and grounded by Retrieval-Augmented Generation (RAG) over an algorithm-theory corpus. Given a task, retrieval assembles a small team; a starter agent writes a C++17 solution; each turn runs public tests in a sandbox, lets the active specialist keep, repair, or hand off the draft, and forwards a structured packet to the next; a final fixer normalizes boilerplate. ³

On the CodeContests test split with Gemma 4, MARS reaches 0.624 ± 0.006 pass rate at 2.3 pipeline stages per task — a +14.4 percentage-point gain over direct prompting — and closes most of the gap to a stronger 0.731 baseline while using 3.3× lower wall-clock time and showing lower per-task token-spend variance. Source code is available on GitHub. ³

PROOF-Gen turns failures into training data for tool use

Tool-calling agents are typically distilled by supervised fine-tuning on teacher-generated trajectories, but standard generate-and-filter pipelines discard failures — and on τ2-bench, 57% of teacher trials fail, with two-thirds being near-misses. PROOF-Gen (Per-scenario Reflective Optimization to Overcome Failed Generation) analyzes each failed scenario’s execution trace and evaluation feedback, writes corrective guidance to steer the teacher to a passing trajectory, then strips that guidance so the student learns from clean demonstrations without task-specific scaffolding. ⁴

Per-scenario optimization recovers 93% of failed scenarios on τ2-bench. Fine-tuning on the combined data lifts Qwen3-4B-Instruct-2507 from Pass^1 0.132 to 0.529, and a Gemma 4 E4B-it model gains +7.2 percentage points on a multi-turn benchmark; in a deployed pipeline, trajectory quality improves by +6.3 points in goal completion and transfers to an on-device model (+1.5 points goal completion and +1.7 to +5.0 across response-quality metrics), with positive transfer in every locale (non-English average +1.48 points). ⁴

The practical message: don’t throw away failures — structure them into prompts that produce gold traces and recycle them into the next distillation cycle to reduce frontier-teacher cost. Further work will likely automate the reflector and verify that improvements persist under independent judges. ⁴

Why It Matters

The common thread is squeezing more out of existing models by improving design, evaluation, and data reuse: explicit experimental design for data mixing, evaluation setups that disclose what’s moving memory scores, specialist relays for tough code tasks, and reflective optimization that turns failed tool calls into usable signal. ¹

For teams with limited compute, this points to doing more with current budgets: better-chosen proxy runs, measurement discipline, specialist prompting, and harvested failures — methods that reallocate spending toward the most promising scale-up runs. ¹