One-Line Summary

High-fidelity medical data curation, smarter quantization, and visual pretraining deliver measurable gains without bigger models.

Research Papers

MedPMC turns PMC into 11M medical image–text pairs

MedPMC is an automated, continuously updatable pipeline that turns permissively licensed PubMed Central (PMC) articles into high‑fidelity medical image–text pairs for training multimodal models. Applied to 6.1 million PMC articles, it curates 11 million image–text pairs suitable for medical foundation models. ¹

Quality controls are central: component evaluations report F1 = 93.2 for initial screening, F1 = 96.5 for multi‑panel figure detection, mean Average Precision (mAP) = 89.8 for figure separation, F1 = 81.4 for caption alignment, and F1 = 96.5 for medical figure classification. A manual review by five annotators (three with medical training) found 95.3% of MedPMC images medically relevant, compared with 19.7% in a prior PMC‑derived dataset — a drastic improvement in dataset cleanliness. ¹

Trained on MedPMC, a Contrastive Language–Image Pretraining (CLIP)‑style model lifts average zero‑shot area under the Receiver Operating Characteristic curve (AUC) by 7.1 percentage points across 26 benchmarks spanning 11 specialties, beating the strongest architecture‑matched biomedical CLIP baseline while using fewer than half as many pairs. Used as the vision encoder inside a multimodal large language model (LLM), it improves medical visual question‑answering by 1.9 and 16.9 points on two benchmarks, and on 10,524 Yale New Haven Health dermatology photos it raises retrieval Recall@5 by 11.7 points. ¹

The authors publicly release the framework, corpus, benchmarks, and pretrained models. For teams building healthcare AI, that means a reproducible path to train and evaluate medical multimodal systems without negotiating direct access to clinical records. ¹

KronQ keeps a 70B model usable at 2-bit weights

KronQ is a post‑training quantization (PTQ) method that compresses large language models (LLMs) without retraining by incorporating gradient covariance alongside activation statistics under a Kronecker‑factored Hessian approximation. Prior second‑order approaches such as GPTQ (a widely used post‑training quantization method) typically rely only on input activations and implicitly treat all output channels equally. ²

KronQ contributes two pieces: bidirectional incoherence processing extends input‑side random rotations to the output dimension using gradient covariance, smoothing weight magnitude variance in both directions; and a new sensitivity metric allocates mixed precision across layers using traces of the activation and gradient Hessians. ²

On 2‑bit weight‑only quantization of LLaMA‑3‑70B, competing methods GPTQ and GPTAQ (another second‑order quantization method) either diverge or yield degenerate results with perplexity above 2000 on WikiText‑2, while KronQ reaches a perplexity of 7.93 — indicating stable ultra‑low‑bit compression where others fail. ²

Visual pretraining shows gains over text-only on document data

This study argues that training directly on visual documents — figures, typeset equations, and page layouts — without first extracting text can teach models more efficiently than text‑only pretraining on the same sources. In a systematic comparison across multiple backbones and benchmarks, unsupervised visual pretraining consistently outperforms text‑only pretraining on the same corpora. ³

The practical idea is to “read the page as an image,” preserving structure that is lost in text extraction. For teams with PDFs, scientific articles, or web pages, this suggests a simpler pipeline that keeps visual context intact rather than converting everything to plain text first. ³

Why It Matters

Medical AI has been constrained by data access and quality; MedPMC shows that high‑fidelity, reproducible curation from open literature can markedly improve image–text modeling (e.g., +7.1 AUC average in zero‑shot CLIP‑style tests) and downstream clinical tasks, without touching sensitive patient records. ¹

On the deployment side, KronQ’s stable 2‑bit weight‑only quantization (7.93 perplexity on LLaMA‑3‑70B) points to cheaper, smaller‑footprint inference — and combined with the document‑image evidence for visual pretraining, it underscores a broader theme: getting more from existing data and models rather than just scaling size. ²

This Week to Try

1. MedPMC overview: Skim the abstract and figures to see how image–text pairs are built and validated. ¹
2. KronQ method tour: Read the method section to understand bidirectional incoherence and why it keeps 2‑bit quantization stable. ²