PagedWeight cuts GPU memory for expert language models by up to 72%

One-Line Summary

Serving-time precision and representation tweaks are squeezing more from today’s compute: dynamic weight quantization for expert models, robot-frame inputs that stabilize control across views, and a Ray update that smooths data pipelines.

Research Papers

PagedWeight compresses expert LLM weights on the fly for memory-bound serving

PagedWeight is a serving-side method that helps expert-based language models fit into limited memory by compressing weights at runtime and adapting to how much memory the key–value cache needs. Mixture-of-Experts (MoE) large language models (LLMs) route tokens to a small set of expert sub-networks for efficiency, but in long conversations the key–value (KV) cache can dominate memory and squeeze out model weights. PagedWeight dynamically quantizes expert weights at runtime and explicitly balances expert precision with KV cache size to keep quality, memory, and latency in check. ¹

In tests across memory-sensitive MoE serving scenarios, the authors report FP16-equivalent accuracy with up to 72.0% graphics processing unit (GPU) memory savings and as much as 1.94× higher throughput. Compared with existing weight-quantization baselines under a similar memory budget, PagedWeight improves output quality by up to 39.3% while incurring at most a 4.1% throughput loss. ¹

The system acts a bit like operating-system paging for precision: when the KV cache grows, it lowers precision for some experts; when memory pressure eases, it raises precision again. For operators running long-context MoE assistants, this exposes a tunable curve between answer quality and serving cost rather than a fixed tradeoff; the paper details how different settings shift that curve under varied workloads. ¹

Robot-centric pointmaps help models follow instructions across camera views

This paper gives instruction-following robots a picture of the scene in the robot’s own coordinate frame so actions line up with what the camera sees even when the viewpoint changes. Vision-Language-Action (VLA) models observe images in the camera frame but must execute in the robot frame, which creates a mismatch that grows as datasets aggregate demonstrations from many camera setups. ²

The method builds images whose pixels store 3D coordinates of scene points in the robot frame, preserving the usual H×W grid so pre-trained 2D VLAs can use them with minimal architectural change. On RoboCasa, pointmaps improve both pi0.5 and SmolVLA and outperform representative camera-viewpoint and 3D-aware baselines; on a real robot, their advantage over an RGB-only policy widens when the camera is moved to an unseen placement. ²

Open Source & Repos

Ray 2.56.1 fixes data conversion regressions and stability issues

Ray is a distributed compute engine for AI workloads; the 2.56.1 release highlights multiple Ray Data fixes around converting datasets to pandas, including an opt-out flag (RAY_DATA_ENABLE_ARROW_BACKED_PANDAS_CONVERSION) for Arrow-backed conversion, a concat crash from int64/double[pyarrow] overflow, and a TensorDtype.from_arrow crash. ³

For teams building training and inference pipelines atop Ray Data, these changes reduce failure risk when handing off to pandas and give a clear escape hatch if Arrow-backed behavior breaks downstream code. If you upgraded to 2.56 and hit regressions, 2.56.1 provides targeted stability without rolling back the series. ³

Why It Matters

Serving-time precision scheduling for expert LLMs (PagedWeight) and robot-frame inputs that keep control robust across views both target the boundary between model capability and system constraints—memory pressure and frame mismatch—so you can keep quality while using the same hardware and data. ¹

On the infrastructure side, surgical fixes in core libraries like Ray prevent pipeline stalls and rollbacks, which often yields more real throughput than adding hardware when teams are bottlenecked on data preparation and conversion. ³