A tiny 'copy‑paste' head makes language models train more efficiently and recall long context

One-Line Summary

Small, targeted changes — a copy-paste head, path-level forecasting, a personalization benchmark, and smarter expert routing — aim to make AI systems more practical in long context, planning, and user-specific settings.

Research Papers

Copy-Paste Language Models add a second head for in-context copying

Rather than always composing the next token from scratch, the model can decide to copy spans directly from what it just read — enabled by a lightweight copy‑paste head added alongside the standard output layer. The authors validate this across large language models (LLMs) by pretraining from 135M up to 1.7B parameters, showing consistent gains in downstream tasks with negligible training and inference overheads. ¹

Beyond raw accuracy, the approach creates useful operating points for long‑context and Retrieval‑Augmented Generation (RAG) scenarios where the best answer is often to quote the source precisely. Reported improvements hold across model scales and backbones, suggesting the copy‑paste head is broadly compatible rather than architecture‑specific. ¹

Why it works: the extra head provides an alternate gradient path that partially mitigates the language modeling (LM) head’s gradient bottleneck, making it easier for the model to learn when to copy versus generate. For practitioners, this points to a simple add‑on to improve retrieval and citation fidelity without materially slowing training or inference. ¹

PathFM generates full future paths instead of point forecasts

Most forecasters return points or quantiles; PathFM generates entire future trajectories so you can reason about cumulative demand, peak load, or event timing — not just single‑step errors. It trains a generative model with flow matching to draw complete paths that reflect dependencies across time, rather than independent per‑step guesses. ²

To make this practical, the authors distill the sampler to one or two steps and share a history attention cache across draws, enabling multiple trajectory samples in roughly the time for one quantile query. PathFM matches strong marginal accuracy on fev‑bench and TIME, introduces a complementary Forecast Path Score Bench (fps‑bench), and achieves a higher aggregate path score than all evaluated external baselines; on chaotic systems, it sustains long‑run variability where strong baselines rapidly collapse. ³

LIBERO-PeRM: benchmarking personalized robotic manipulation

This benchmark tests how embodied agents learn and respect a user’s preferences while still completing the task, separating “did it work?” from “was it done the way I like?”. LIBERO‑PeRM defines language‑expressible preferences across motion, skill, and user‑profile choices, provides paired demonstrations with and without preferences, and packages five task suites (312 tasks) with automatic evaluation and reusable preferences. ⁴

Experiments find current vision‑language‑action (VLA) policies struggle at zero‑shot personalization; supervised fine‑tuning (SFT) makes preferences learnable. Mixed training retains no‑preference performance better than sequential adaptation and improves results with contextual prompts compared with preference‑only training, but can weaken motion‑preference learning. Generalization, preference composition, and conflict handling remain open challenges. ⁴

MirrorMoE rethinks exploration in mixture-of-experts routing

Mixture of Experts (MoE) models scale by activating only a few experts per token. Modern designs often combine fine‑grained expert segmentation (more combinations at a fixed activation ratio) with shared experts (always active), but this pairing dampens routing exploration because shared experts reduce representation variability — precisely when more exploration is needed. ⁵

MirrorMoE tackles the dilemma by shrinking the exploration space with token‑adaptive suppression of unnecessary experts, reconciling the need to explore with the tendency of shared experts to anchor routing. The paper reports consistent improvements across model scales, indicating better use of MoE capacity without architectural upheaval. ⁵

Open Source & Repos

cmux: a Ghostty-based macOS terminal for agent-heavy coding

cmux is a macOS terminal built on the Ghostty renderer with vertical tabs and notifications, tailored for juggling multiple AI coding agents and long‑running jobs. It emphasizes multitasking, organization, and programmability so developers can monitor agents, logs, and tools side‑by‑side. ⁶

A prerelease dated Oct 18, 2026 packages a binary XCFramework (universal macOS arm64/x86_64) for the app’s remote‑desktop core and layout reducer, with a Swift Package Manager binary‑target checksum provided for integration. Teams experimenting with agent workflows in the terminal may find the vertical‑tab and notification model especially helpful. ⁶

Why It Matters

Today’s papers target hidden bottlenecks — output‑head training dynamics, path‑level uncertainty, and preference disentanglement — with small, comprehensible changes. Copy‑Paste Language Models offer a low‑overhead way to improve quoting and retrieval fidelity in long context, while PathFM’s full‑trajectory generation better supports path‑dependent decisions like cumulative demand and peak load. ²

For builders, the combination of a personalization benchmark (LIBERO‑PeRM), more reliable MoE routing (MirrorMoE), and agent‑friendly tooling (cmux) points toward AI systems that are not just bigger, but easier to steer and operate. These are practical steps teams can adopt or benchmark against today. ⁶

Try This Week

1. cmux on macOS: Download from the GitHub repo (Releases tab) and try vertical tabs to monitor multiple agent runs in one window. ⁶