[feat] Initial support for VLMs, add Qwen2.5VL GRPO example

[hiyouga]

What does this PR do?

This PR migrates the feature of RL on VLMs in our implementation in EasyR1 fork back to veRL. We have validated this feature using Qwen2.5-VL 7B model on 8*H100 GPUs. The configuration and data processing script are provided along this PR for easy reproducing.

How to reproduce?

1. Download and preprocess the dataset

python3 examples/data_preprocess/geo3k.py --local_dir ~/data/geo3k

2. Start GRPO training

bash examples/grpo_trainer/run_qwen2_5_vl-7b.sh

Dependencies

• vllm>=0.7.3
• transformers>=4.49.0
• qwen-vl-utils
• mathruler

Major Changes

• DataProto / Sharding Manager
  • Support list of tensors in non-tensor batch (we need this to carry the image features in varied length)
• Actor/Ref/Critic
  • Add image features to fsdp model inputs
  • Process position ids to adapt to the m-rope
• Rollout
  • Update dtensor weight loader to adapt to the Qwen2-VL architecture in vLLM 0.7+
  • Add image features to vllm inputs
  • Use raw_prompt_ids as the vLLM inputs to avoid unpadding the input ids
• Reward Manager
  • Add mathruler for more accurate math scores on the Geometry 3k dataset
• Models
  • Support calculating the position ids for the m-rope in Qwen2-VL
  • Support removing padding in flash attention2 for m-rope (transformers itself did not support it)
• Dataset
  • Support pre-processing the Geometry3k dataset
  • Support processing the images column using the image processor if this column exists
• FSDP Workers / Checkpoint Merger
  • Support AutoModelForVision2Seq at model initialization

Note: the ulysses parallelism is not completed yet. We will support it soon in next update.

Performance

We provide the estimated MFU of the language model part for H100 GPUs. These values are lower than the actual ones since we did not compute the FLOPs of the vision tower part.

• remove_padding=False: MFU ~7%
• remove_padding=True: MFU ~20%

The training and test reward score curves are presented as follows.

Who can review?

@vermouth1992 @PeterSH6

[CLAassistant]

All committers have signed the CLA.

[khazic]

good job

[vermouth1992]

Item TODO: I guess it's better to provide a documentation on how to extend to other VLMs like ulysses/rmpad, etc,. so that other contributors can add their interested VLMs.

[hiyouga]

@vermouth1992 Sure, but it requires non-trivial effort to build a unified guideline for all the VLMs, so we are considering improving the document next PR.