Qwen2.5-VL Technical Report

Concepts: vision-language-models | multimodal-embeddings | vision-transformer | patch-embeddings | visual-grounding Builds on: an-image-is-worth-16x16-words | blip-2-bootstrapping-language-image-pretraining | llava-visual-instruction-tuning | rope-rotary-position-embedding

By 2025 the open VLM stack had stabilized into a familiar recipe: take a CLIP-style vision encoder pretrained at a fixed input resolution (say 336x336), bolt it to a pretrained LLM via a thin projection layer, train on instruction data. Qwen2.5-VL keeps the structural skeleton but rebuilds the vision side from first principles, letting the encoder consume images at their native resolution and videos as long as several hours, with grounding accurate enough to drive a GUI agent.

The core idea

The analogy: Most prior VLMs are like a security guard who can only watch a single camera at a fixed zoom. Want to look at the parking lot, the lobby, and a whiteboard close up? Crop or downscale each one to fit the same screen. Qwen2.5-VL is the guard who walks up to whatever camera is needed, at whatever zoom the situation calls for, and remembers when each thing happened down to the second.

Three changes do the heavy lifting:

1. Native dynamic resolution. Instead of forcing a 336x336 (or 448, or 672) crop, the ViT processes the image at whatever resolution it arrives in. Number of visual tokens scales with image area. A document page becomes thousands of tokens; a thumbnail becomes a few dozen.
2. Window attention in the encoder. Native resolution means quadratic attention on tens of thousands of patches would be lethal. Most ViT blocks use windowed attention (constant cost per window), with only a handful of blocks doing global attention to mix information across the whole image.
3. Absolute time encoding (M-RoPE extended). Video frames carry a real timestamp, not just a frame index. The model can answer “what happens at 2:13?” because position embeddings encode wall-clock time, not relative ordering.

“Qwen2.5-VL introduces dynamic resolution processing and absolute time encoding, enabling it to process images of varying sizes and videos of extended durations (up to hours) with second-level event localization.”

The grounding capability falls out for free: because the model preserves spatial geometry through native-resolution processing, it can emit pixel-accurate bounding boxes and points, in JSON, without any post-hoc OCR pipeline.

What’s clever — find the instinct

The non-obvious move is treating the visual encoder as a first-class object that gets trained from scratch alongside the LLM, rather than freezing a pretrained CLIP backbone. Most prior open VLMs (LLaVA, BLIP-2, MiniGPT) inherit the CLIP encoder’s resolution baggage because retraining a vision model is expensive and most teams couldn’t justify the compute.

Qwen2.5-VL’s authors argue (and demonstrate) that the gains from native resolution outweigh the cost of training a fresh ViT. The encoder uses RMSNorm and SwiGLU to match the LLM-side conventions, simplifying the pipeline. Window attention then keeps the cost tractable: 7 of every 8 transformer blocks are windowed, only the final block in each group does global mixing.

“By training a native dynamic-resolution Vision Transformer (ViT) from scratch and incorporating Window Attention, we reduce computational overhead while maintaining native resolution.”

The second clever move is the absolute time encoding for video. Earlier VLMs (Video-LLaVA, etc.) used uniform frame sampling with a learnable per-frame position embedding. That tells the model “this frame came after that one” but not “this frame happened at minute 17.” Qwen2.5-VL extends Multimodal RoPE so the temporal axis carries actual seconds, letting the model answer queries that require event localization in time, not just ordering.

Walkthrough: how a 1280x720 screenshot gets processed

Consider feeding Qwen2.5-VL a screenshot of a desktop UI at 1280x720 pixels and asking it to click a specific button.

Step 1: Patchify at 14x14 (no resize)
        1280 / 14 = 91.4, rounded to 91 patches wide
         720 / 14 = 51.4, rounded to 51 patches tall
        Total: 91 x 51 = 4,641 patches

Step 2: Adjacent 2x2 patches merged into single visual token
        4,641 / 4 ≈ 1,160 visual tokens

Step 3: Each token gets a 2D positional embedding
        (token_x, token_y) reflecting actual screen coordinates

Step 4: ViT processes through 32 blocks
        - 28 blocks use window attention (window size 112)
        - 4 blocks do full attention to mix globally
        Cost dominated by the windowed blocks: O(N * w^2) not O(N^2)

Step 5: Output 1,160 vision tokens get projected into LLM space

Step 6: Concatenate with text prompt
        "<image_tokens> Click the Save button."
        and feed to the LLM

Step 7: LLM emits JSON: {"action":"click","point":[842,156]}

Compare to a fixed-resolution VLM: it would resize the screenshot to 336x336, losing pixel-level detail. The Save button might shrink to 4x4 pixels, becoming impossible to localize. Qwen2.5-VL keeps the original 32x12 button intact.

Does it work? What breaks?

Headline numbers from the paper (72B model unless noted):

Benchmark	Qwen2.5-VL-72B	GPT-4o	Claude 3.5 Sonnet
DocVQA (val)	96.4	91.1	95.2
ChartQA	89.5	85.7	90.8
MMMU (val)	70.2	70.3	70.4
RefCOCO (avg)	92.7	—	—
Video-MME (no subs)	73.3	71.9	60.0

DocVQA at 96.4 effectively saturates the dataset. Document and chart understanding are where the native-resolution bet pays off most: small text and numerical labels survive into the encoder.

What breaks:

• The 3B model loses non-trivially on grounding tasks; the receptive field of a small ViT struggles with high-resolution input.
• Window attention introduces a tradeoff: when objects span window boundaries, only the global-attention blocks can stitch them back together. Tasks involving long thin objects (e.g., reading a multi-line address that wraps) can suffer.
• Video understanding above ~30 minutes still degrades; “up to hours” is technically true (you can pass the tokens) but accuracy on questions requiring fine-grained recall drops sharply past 1 hour.
• The grounding output format (JSON with bounding boxes) is brittle to prompt formatting; small changes to how you ask cause large changes in coordinate accuracy.

So what?

For a practitioner running a VLM in production for document or screenshot understanding, this paper rewires several priors:

1. Stop downsizing inputs to fit the encoder. If your task involves text-on-image or fine-grained UI elements, resolution is the dominant factor. A model that natively handles 2K+ images outperforms a much larger model that takes 448x448 crops.
2. Visual grounding is now a first-class capability. You don’t need a separate detector. Ask the VLM to emit JSON bounding boxes directly. This collapses VLM + OCR + bbox-regressor pipelines into one call.
3. Video models are not yet trustworthy beyond ~30 minutes. The “hours” claim is real for narrow tasks; broad understanding still requires chunking.
4. For agentic use cases (operating computers, mobile devices), Qwen2.5-VL is the open baseline. The model card explicitly trains on UI traces, and the absolute spatial coordinates are predicted in pixel space, not normalized [0,1].

“Qwen2.5-VL excels not only in static image and document understanding but also as an interactive visual agent capable of reasoning, tool usage, and task execution in real-world scenarios such as operating computers and mobile devices.”

For Saikat’s stack specifically: any pipeline that crops images to a fixed input is leaving accuracy on the table. The native-resolution approach is the right default for any document parsing, OCR-adjacent, or UI-grounding workload.

Connections

• vision-language-models — Qwen2.5-VL is the current open-weight VLM benchmark
• vision-transformer — the encoder is a from-scratch native-resolution ViT
• patch-embeddings — non-square dynamic patch counts based on input dimensions
• multimodal-embeddings — projected ViT outputs share LLM token space
• visual-grounding — pixel-coordinate bounding box emission as native output
• long-context — absolute-time M-RoPE handles hour-scale video
• an-image-is-worth-16x16-words — original ViT design that this rebuilds
• llava-visual-instruction-tuning — earlier projection-based VLM architecture
• blip-2-bootstrapping-language-image-pretraining — bridges vs. native vision encoders
• rope-rotary-position-embedding — RoPE is extended to multimodal axes here

Citation

arXiv:2502.13923

Bai, S., Chen, K., Liu, X., et al. (2025). Qwen2.5-VL Technical Report. arXiv preprint. https://arxiv.org/abs/2502.13923