One-Line Summary

Today's papers sharpen how we evaluate and steer AI: a physics-aware test for video generation, evidence that visible "thinking" rarely adds safety, a measurement showing agents drop plans without context, and a human-grounded path to privacy alignment.

Research Papers

Physics Question Scene Graph checks physical realism in AI video

This work introduces a new way to check whether AI-generated videos both follow the prompt and obey basic physical laws. It builds a hierarchical, graph-based question pipeline with a vision-language model (VLM) guided by in-context examples, so each question depends logically on earlier ones and the system can pinpoint which object, action, or relation violates physics. ¹

To validate the method, the authors create FinePhyEval, a dataset of physics-centered prompts and generated videos from state-of-the-art systems including Sora 2, Veo 3, and Wan 2.1, with human annotations across multiple categories. Using this set, PQSG’s fine-grained scores correlate better with human judgments than prior approaches, and it ranks closed-source models higher than Wan 2.1 on physical realism. ¹

They also benchmark two strong VLMs on generating and answering the evaluation questions: the models can write human-like questions but still underperform humans on answering them, highlighting gaps in physical reasoning from video. ¹

For teams building text-to-video, this offers an actionable rubric that localizes failures instead of a single realism score; watch whether vendors adopt PQSG-style question graphs and whether FinePhyEval becomes a common baseline for physics plausibility. ¹

Thinking tokens show little true safety deliberation

This study tests whether the "deliberation" traces used by modern reasoning models actually make them safer. Across open-weight families including GPT-OSS, Qwen, Olmo, and Phi, a simple classifier trained on the first token’s hidden state predicts refusal vs. compliance with 0.84–0.95 area under the receiver operating characteristic curve (AUROC) and about 88% balanced accuracy—before any visible thinking—while the final outcome rarely changes after the first ~20% of thinking; about 74% of text-level “deliberations” occur when the response distribution is already locked. ²

The authors also report that many inference-time and training-based safety interventions shift behavior toward over-refusal and further suppress scarce deliberation signals. The takeaway: current “thinking” often behaves like prefix completion, not internal debate; teams should audit early hidden-state decisions rather than rely on longer, visible reasoning to improve safety. ²

LLM agents drop plans unless kept in context

This paper measures whether agents truly carry a plan forward as internal state or just rely on the plan staying in the context window. Using “replay pairing,” the authors show that on Llama-3.1-70B, plan signal spikes to 0.453 one step after the plan, then falls 4.1× in a single action–observation step; on HotpotQA it drops 12.4×—evidence that large language model (LLM) agents do not persist plans and instead depend on the plan remaining in context. ³

A Llama-trained probe transfers to DeepSeek-R1-Distill-Llama-70B at AUROC 0.748 (p=6e-4), while R1-specific probes reach 1.000, suggesting different hidden-state directions for plan encoding; in practice, naive plan eviction cuts ALFWorld success by 34.7 percentage points, and a probe-gated resurfacing strategy does not recover it. The authors argue context management is load bearing and that plan protection alone is insufficient. ³

PrivacyAlign grounds agent privacy in human judgments

This project centers human judgment to teach agents when to share or withhold information. PrivacyAlign releases a dataset of 1,350 samples with 3,516 detailed annotations from 599 annotators across scenarios where current models actually leak, and shows that conditioning LLM judges on human annotations and explanations makes their evaluations more reliable. ⁴

The authors introduce annotation-conditioned reward modeling for reinforcement learning (RL): the human annotations score new responses during training. Small open-weight agents trained with this reward align better with human privacy norms, with strong gains on PrivacyAlign and existing privacy benchmarks for agents. ⁴

Why It Matters

Evaluations are getting more diagnostic: instead of a single realism score, PQSG localizes where physics breaks; and instead of assuming more visible reasoning increases safety, evidence shows the refusal/compliance decision is often set at the first token’s hidden state. That shifts safety work toward earlier internal signals and targeted tests. ²

For agents, infrastructure is the product: plan memory decays quickly without careful context management, and naive plan eviction can slash task success. At the same time, privacy alignment grounded in human norms points to training recipes that better match user expectations. ³

This Week to Try

1. Build a physics checklist for your next AI video: track shadows, occlusion, momentum, and object interactions as you review clips.
2. Compare “think step by step” vs. short answers in your chatbot on a borderline-safe prompt; note whether the refusal/compliance choice actually changes.