Title: Towards Comprehensive Basketball Understanding
Author: Yirong Hu, Jiayuan Rao, Yu Zhang, Shangzhe Di, Weidi Xie
Institution: School of Mathematical Sciences, Peking University; School of Artificial Intelligence, Shanghai Jiao Tong University; arXiv preprint (cs.CV)
Publication Date: August 24, 2026 (arXiv v1)
The Problem: Great at Reading the Scorebug, Lost on the Possession
General-purpose AI models can read a broadcast scorebug almost perfectly. Ask them who took the shot, where it came from and what that player averages, and they fall apart. Understanding a single possession means recognizing the event, tying it to the right player, placing it on the court and in time, and connecting it to rosters and stats. Existing benchmarks test those abilities one at a time, so little was known about how models perform when skills must be chained. For teams evaluating video AI vendors, that gap is exactly where tagging errors hide.
Methodology: A Specialist Staff, Not One Generalist
The authors built BasketballBench: 7,980 questions across ten tasks, drawn from the 2025-26 NBA season using official play-by-play, profiles for 530 active players and 2,501 possession-level broadcast clips. They then built BasketballSkills, an agent that chains specialized tools.
Eight atomic tools (the specialist staff): Face recognition, jersey reading, scorebug reading, player and ball tracking, event detection, shot-zone classification, event timing and a stats database lookup each do one job, like position coaches.
Four procedural skills (the playbook): Written workflows set which tools run and in what order, like a set play that calls out each action.
Language-model controller (the head coach): DeepSeek-V4-Flash reads the question, picks a skill and calls tools step by step.
Verifier (the replay review): A simple program checks every tool call before it runs.
Why it Matters: Chaining Skills Is Where AI Breaks
Player grounding is the bottleneck: GPT-5.4 read jersey numbers from player crops at 92.3%, but its full play-by-play F1 was just 29.8%, versus 59.6% when only event types had to be right. Boxing the shooter on screen lifted shooter identification by 19.7 points on average across models.
Composed skills win big: BasketballSkills led eight of ten tasks, including 99.2% versus 40.2% on game-specific questions, 83.7% versus 53.0% on player knowledge from video, and 45.9 versus 29.8 play-by-play F1.
Not solved yet: It trailed GPT-5.4 on event timing (47.8% versus 77.5%) and on scorebug reading, scored 69.8% on shot zones with frequent paint and restricted-area mixups, and still misses most full play-by-play events. This is a preprint, and the video comes from 33 games.
ACTIONABLE TAKEAWAYS
Test vendors on chained questions: Ask who committed the foul and what he averages, not just what number he wears.
Wire video AI to your own data: The knowledge gains came from querying structured records, not model memory. Connect tagging tools to your roster and play-by-play feeds.
Keep humans on timing and paint calls: Spot-check clip timestamps and restricted-area versus paint labels before trusting automated shot charts.
Research Paper Link: Towards Comprehensive Basketball Understanding