Accuracy on the VideoMathQA benchmark using direct answer format (no CoT reasoning) for MCQ and Multi-Binary tasks, with and without subtitles.
Shows model performance across mathematical concepts and video lengths. This leaderboard is sorted by results on Multi-Binary with subtitles.
# | Models | Size | MCQ | MBin | Mathematic Concepts | Duration | |||||||||||||
---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
V | +Sub | V | +Sub | GAng | GAre | GLen | Chart | Stat | Arth | Topo | Grph | Cntg | Pzle | Short | Med | Long | |||
Claude-3.7-sonnet | - | 26.2 | 27.1 | 8.6 | 9.5 | 17.4 | 9.9 | 5.9 | 8.0 | 17.4 | 11.5 | 13.3 | 5.6 | 5.3 | 9.3 | 8.2 | 11.0 | 9.1 | |
GPT-4o | - | 20.2 | 24.5 | 12.6 | 13.6 | 13.0 | 12.7 | 15.7 | 12.0 | 4.4 | 17.3 | 20.0 | 5.6 | 7.9 | 20.4 | 14.2 | 15.6 | 10.6 | |
Gemini-2.0-Flash | - | 28.6 | 31.7 | 14.1 | 20.5 | 30.4 | 23.9 | 27.5 | 13.3 | 8.7 | 19.2 | 13.3 | 16.7 | 7.9 | 33.3 | 25.4 | 24.0 | 11.4 | |
Gemini-1.5-Flash | - | 20.5 | 23.1 | 12.6 | 17.6 | 26.1 | 15.5 | 19.6 | 9.3 | 17.4 | 23.1 | 6.7 | 22.2 | 15.8 | 24.1 | 17.9 | 22.1 | 12.1 | |
Qwen2.5-VL | 3B | 26.9 | 27.6 | 19.3 | 19.6 | 26.1 | 23.9 | 23.5 | 21.3 | 34.8 | 17.3 | 26.7 | 11.1 | 15.8 | 20.4 | 25.4 | 23.4 | 15.9 | |
InternVL2.5 | 2B | 24.3 | 20.7 | 14.3 | 14.5 | 21.7 | 9.9 | 27.5 | 10.7 | 4.4 | 15.4 | 20.0 | 0.0 | 15.8 | 16.7 | 17.9 | 16.9 | 8.3 | |
PLM-LLaMA | 3B | 22.9 | 22.1 | 13.6 | 15.0 | 17.4 | 16.9 | 25.5 | 8.0 | 26.1 | 9.6 | 20.0 | 11.1 | 13.2 | 13.0 | 16.4 | 18.8 | 9.1 | |
InternVL3 | 2B | 22.4 | 23.3 | 18.8 | 16.4 | 21.7 | 16.9 | 17.7 | 17.3 | 30.4 | 15.4 | 20.0 | 22.2 | 13.2 | 5.6 | 18.7 | 14.9 | 15.9 | |
PLM-LLaMA | 8B | 22.1 | 23.1 | 16.7 | 14.5 | 13.0 | 11.3 | 17.7 | 13.3 | 17.4 | 17.3 | 20.0 | 11.1 | 10.5 | 16.7 | 16.4 | 14.9 | 12.1 | |
Oryx-1.5 | 7B | 22.6 | 22.6 | 16.9 | 17.4 | 13.0 | 23.9 | 23.5 | 9.3 | 21.7 | 23.1 | 20.0 | 5.6 | 18.4 | 11.1 | 20.2 | 20.8 | 10.6 | |
LLaVA-OV | 7B | 20.7 | 21.2 | 14.8 | 15.5 | 8.7 | 15.5 | 17.7 | 16.0 | 30.4 | 17.3 | 13.3 | 5.6 | 15.8 | 11.1 | 16.4 | 18.8 | 10.6 | |
LongVA-DPO | 7B | 21.4 | 21.7 | 16.2 | 14.1 | 8.7 | 15.5 | 17.7 | 12.0 | 30.4 | 9.6 | 6.7 | 5.6 | 10.5 | 18.5 | 14.9 | 11.7 | 15.9 | |
Video-R1 | 7B | 21.4 | 17.4 | 16.0 | 16.2 | 8.7 | 22.5 | 25.5 | 16.0 | 26.1 | 13.5 | 6.7 | 5.6 | 13.2 | 9.3 | 16.4 | 16.9 | 15.2 | |
InternVL2.5 | 8B | 24.3 | 24.8 | 18.6 | 18.6 | 26.1 | 19.7 | 17.7 | 17.3 | 21.7 | 19.2 | 26.7 | 11.1 | 10.5 | 20.4 | 17.9 | 22.7 | 14.4 | |
LLaVA-Video | 7B | 26.9 | 26.4 | 20.0 | 19.3 | 13.0 | 21.1 | 31.4 | 17.3 | 17.4 | 15.4 | 26.7 | 5.6 | 18.4 | 18.5 | 23.9 | 20.8 | 12.9 | |
InternVideo2.5 | 8B | 25.2 | 28.6 | 19.1 | 19.1 | 34.8 | 22.5 | 15.7 | 14.7 | 21.7 | 19.2 | 20.0 | 27.8 | 10.5 | 18.5 | 18.7 | 22.1 | 15.9 | |
Qwen2.5-VL | 7B | 26.7 | 27.9 | 19.8 | 19.1 | 8.7 | 25.4 | 25.5 | 18.7 | 13.0 | 23.1 | 13.3 | 5.6 | 15.8 | 16.7 | 22.4 | 19.5 | 15.2 | |
InternVL3 | 8B | 29.1 | 27.9 | 20.0 | 20.7 | 13.0 | 29.6 | 27.5 | 13.3 | 13.0 | 28.9 | 20.0 | 22.2 | 15.8 | 14.8 | 25.4 | 24.0 | 12.1 | |
VideoChat-R1 | 7B | 27.6 | 29.1 | 21.2 | 21.2 | 8.7 | 22.5 | 31.4 | 21.3 | 17.4 | 30.8 | 6.7 | 11.1 | 15.8 | 18.5 | 26.9 | 20.1 | 16.7 | |
Aria | 34B | 23.8 | 26.4 | 17.4 | 19.1 | 8.7 | 25.4 | 19.6 | 22.7 | 17.4 | 19.2 | 20.0 | 11.1 | 21.1 | 11.1 | 21.6 | 16.9 | 18.9 | |
Oryx-1.5 | 32B | 30.5 | 33.1 | 22.9 | 24.1 | 30.4 | 39.4 | 31.4 | 10.7 | 17.4 | 21.2 | 6.7 | 11.1 | 15.8 | 33.3 | 27.6 | 29.9 | 13.6 | |
Qwen2.5-VL | 32B | 32.4 | 32.6 | 25.7 | 24.8 | 43.5 | 31.0 | 25.5 | 14.7 | 26.1 | 26.9 | 6.7 | 27.8 | 10.5 | 33.3 | 28.4 | 30.5 | 14.4 | |
InternVL2.5 | 38B | 31.0 | 33.6 | 24.1 | 26.0 | 43.5 | 38.0 | 39.2 | 8.0 | 13.0 | 32.7 | 6.7 | 11.1 | 18.4 | 29.6 | 34.3 | 31.8 | 10.6 | |
InternVL3 | 38B | 31.7 | 35.7 | 25.2 | 29.5 | 34.8 | 42.3 | 37.3 | 13.3 | 17.4 | 25.0 | 13.3 | 33.3 | 26.3 | 40.7 | 35.8 | 38.3 | 12.9 | |
LLaVA-Video | 72B | 28.3 | 30.0 | 20.2 | 24.3 | 8.7 | 32.4 | 25.5 | 20.0 | 13.0 | 36.5 | 13.3 | 22.2 | 21.1 | 24.1 | 27.6 | 27.3 | 17.4 | |
LLaVA-OV | 72B | 25.5 | 28.3 | 21.0 | 24.8 | 17.4 | 31.0 | 23.5 | 12.0 | 21.7 | 38.5 | 20.0 | 27.8 | 18.4 | 31.5 | 30.6 | 28.6 | 14.4 | |
InternVL2.5 | 78B | 33.3 | 31.7 | 28.3 | 27.9 | 39.1 | 36.6 | 31.4 | 18.7 | 26.1 | 32.7 | 26.7 | 27.8 | 13.2 | 27.8 | 33.6 | 35.1 | 13.6 | |
Qwen2.5-VL | 72B | 36.9 | 37.6 | 26.0 | 27.9 | 26.1 | 36.6 | 31.4 | 17.3 | 30.4 | 38.5 | 20.0 | 16.7 | 18.4 | 29.6 | 34.3 | 29.2 | 19.7 | |
InternVL3 | 78B | 33.3 | 31.7 | 28.3 | 27.9 | 39.1 | 36.6 | 31.4 | 18.7 | 26.1 | 32.7 | 26.7 | 27.8 | 13.2 | 27.8 | 33.6 | 35.1 | 13.6 |