Warriors at Knicks · Madison Square Garden · December 14, 2021
A
B
C
D
Which one is real?
A · MSL's neural rendering model
B · A video game
C · The real broadcast
D · A text-to-video model
Our neural rendering model is as believable as the real broadcast.
The problem
How do you generate believable basketball games, live?
Vision
Simulation
Rendering
Captures every player's movement in 3D from game broadcasts.
Learns each player's style and generates new games.
Shows the new games as broadcast video, live.
Nobody has done this. Video games script their players by hand. Text-to-video models can't hold a game together.
Vision reconstructs games in 3D
from 2D broadcasts.
from 2D broadcasts.
The camera, the ball, and every player.
The ball, 14 feet above the floor
found on 99 frames in 100
The camera, its aim measured to a third of a degree
The ball, followed through every frame
Every player, tracked and named
The broadcast becomes the game in 3D.
Every body at its real size · the ball's flight in 3D
MSL's vision pipeline
- Camera SolverFOCALOurs
- Player TrackerMeta's SAM 3.1Open
- Identity Labelera vision-language model, checked against the official rosterAgent
- Human Mesh ReconstructionMeta's SAM 3D BodyOpen
- Body PlacementMC-HMROurs
- Ball TrackerBLOT and BLOT3DOurs
The simulation generates the game, half a second at a time.
The STAGE Transformer · about 400 million parameters, trained from scratch
Every player, the ball, the camera,
and the game's state is a row.
Every half second is a column.
and the game's state is a row.
Every half second is a column.
Each player attends back
along their own track.
along their own track.
Then every row attends across the step,
and the next half second is generated.
and the next half second is generated.
It learned how games unfold
from 38,000 games of play-by-play,
and how players move from game broadcasts.
from 38,000 games of play-by-play,
and how players move from game broadcasts.
Each player plays in their real style:
how they move, where they shoot from,
and the choices they make.
how they move, where they shoot from,
and the choices they make.
A rules engine keeps every play legal.
Within the rules, no two generated
games are the same.
Within the rules, no two generated
games are the same.
Rendering turns the 3D game
into broadcast video.
into broadcast video.
The renderer is told, every frame
- how far away every surface is
- which way every surface faces
- which person each pixel belongs to
- what every surface looks like
The simulation decides what happens.
The renderer makes it believable.
The renderer makes it believable.
What PAINT, our neural renderer, draws from that alone
It holds together as long as the game lasts.
our renderer, two minutes into one continuous take
the real broadcast of that moment
We set the court, the arena, and the roster.
The branding and the jerseys are ours to set as well, and every player keeps their own look.
Jordan · O'Neal · Iverson · Bryant · Garnett · James · Durant · Jokić · Leonard · Curry
The look is ours to control, down to every logo.
Oracle Arena
Madison Square Garden
TD Garden
United Center
Progress so far
The research is done. Next, from the lab to the living room.
Fourteen research papers push the frontiers of vision, simulation, and rendering forward.
- Vision
- Recovers the whole game in 3D from one broadcast camera.camera aim within a third of a degree · the ball on 99 frames in 100 · every body at its real size
- Simulation
- Plays whole games under the rules, each player in their own style.about 400 million parameters · trained from scratch
- Rendering
- Renders a game as long as it lasts, in any arena.an open five-billion-parameter video backbone · trained on 3 million frames of real broadcast
The platform · about 770,000 lines of code · 49 packages · Python, TypeScript, and GPU shaders
Roadmap
Next, we complete the broadcast engine
and scale up data extraction and training.
and scale up data extraction and training.
Every camera
The wide shot, the baseline, the close-up, cut by a camera operator, for the whole game.
Live streaming
A simulated game rendered as fast as it plays, streamed as a live broadcast.
More data
Every game ever broadcast, extracted and trained on. Every model improves with more data and more training.
Any matchup
Any game you can describe. Jordan's Bulls against Curry's Warriors, at the Garden.
Sharper video
A larger open-weight video backbone, for sharper, more detailed broadcast video.
Every sport
The same approach extends to soccer, football, and every team sport played with a ball.