Physics Simulation in AI Video: How Models Finally Learned to Respect Reality

The Basketball Problem

OpenAI described it perfectly when launching Sora 2: in earlier models, if a basketball missed the hoop, it would simply materialize inside the net anyway. The model knew the narrative outcome (ball goes in basket) but had no concept of the physical constraints that should govern the journey.

This was not a minor bug. It was symptomatic of a fundamental architectural limitation. Early video generation models excelled at visual pattern matching, learning to generate frames that looked individually plausible while remaining physically incoherent when viewed in sequence.

Three Pillars of Physical Understanding

The breakthrough in physics simulation rests on three interconnected advances: world modeling, chain-of-thought reasoning, and improved temporal attention mechanisms.

World Models vs Frame Prediction

Traditional video generation treated the task as sequential frame prediction: given frames 1 through N, predict frame N+1. This approach inherently struggles with physics because it has no explicit representation of the underlying physical state.

World models take a fundamentally different approach. Instead of predicting pixels directly, they first construct an internal representation of the scene's physical state, including object positions, velocities, materials, and interactions. Only then do they render this state into visual frames. This approach, explored in depth in our world models analysis, represents a paradigm shift in how we think about video generation.

Chain of Thought for Video

Kling O1, released in late 2025, introduced chain-of-thought reasoning to video generation. Before generating frames, the model explicitly reasons about what should physically happen in the scene.

For a scene of a glass falling off a table, the model first reasons:

• Glass has initial velocity zero, position on table edge
• Gravity accelerates the glass downward at 9.8 m/s²
• Glass contacts floor after approximately 0.45 seconds
• Glass material is brittle, floor is hard surface
• Impact exceeds fracture threshold, glass shatters
• Shards scatter with momentum conservation

This explicit reasoning step happens in the model's latent space before any pixels are generated. The result is video that respects not just visual aesthetics but causal chains.

Temporal Attention at Scale

The architectural foundation enabling these advances is temporal attention, the mechanism by which video models maintain consistency across frames. The diffusion transformer architecture that powers modern video models processes video as spacetime patches, allowing attention to flow both spatially within frames and temporally across them.

Modern video models process millions of spacetime patches per video, with specialized attention heads dedicated to physical consistency. This scale allows models to track object identity and physical state across hundreds of frames, maintaining coherence that was impossible with earlier architectures.

Real-World Physics Benchmarks

How do we actually measure physics simulation quality? The field has developed several standardized tests:

Kling models have consistently excelled at fluid dynamics, with particularly impressive water simulation and cloth physics. OpenAI's Sora 2 leads in collision realism and momentum conservation, handling complex multi-object interactions with impressive accuracy.

The Gymnast Test

One of the most demanding physics benchmarks involves Olympic gymnastics. A tumbling gymnast undergoes complex rotational dynamics: angular momentum conservation, variable moment of inertia as limbs extend and contract, and precise timing of force application for takeoffs and landings.

Early video models would generate impressive individual frames of gymnasts in mid-air but fail catastrophically on the physics. Rotations would speed up or slow down randomly. Landings would occur at impossible positions. The body would deform in ways that violated anatomical constraints.

Sora 2 explicitly highlighted Olympic gymnastics as a benchmark it now handles correctly. The model tracks the gymnast's angular momentum through the entire routine, accelerating rotation when limbs pull in (ice skater spin effect) and decelerating when they extend.

Material Understanding

Physics simulation extends beyond motion to material properties. How does a model know that glass shatters while rubber bounces? That water splashes while oil pools? That metal deforms plastically while wood snaps?

The answer lies in the training data and the model's learned priors. By training on millions of videos showing materials interacting with the world, models develop implicit material understanding. A glass falling on concrete produces a different outcome than glass falling on carpet, and modern models capture this distinction.

Limitations and Edge Cases

Despite these advances, physics simulation in AI video remains imperfect. Several known limitations persist:

Long-term stability: Physics remains accurate for 5-10 seconds but can drift over longer durations. Extended videos may gradually violate conservation laws.

Complex multi-body systems: While two objects colliding works well, scenes with dozens of interacting objects (like a falling Jenga tower) can produce errors.

Unusual materials: Training data biases mean common materials (water, glass, metal) simulate better than exotic ones (non-Newtonian fluids, magnetic materials).

Extreme conditions: Physics at very small scales (molecular), very large scales (astronomical), or extreme conditions (near light speed) often fail.

Implications for Creators

What does improved physics simulation mean for video creators?

First, it dramatically reduces the need for post-production fixes. Scenes that previously required careful editing to correct physical impossibilities now generate correctly the first time.

Second, it enables new creative possibilities. Accurate physics simulation means Rube Goldberg machines, sports sequences, and action scenes can be generated without painstaking manual correction.

Third, it improves viewer perception. Viewers subconsciously detect physics violations, making physically accurate videos feel more real even when the difference is hard to articulate.

The Road Ahead

Physics simulation will continue to improve along several axes:

Longer temporal consistency: Current models maintain physics for seconds, future models will maintain it for minutes.

More complex interactions: Scenes with hundreds of interacting objects will become feasible.

Learned physics engines: Rather than implicit physics from training data, future models may incorporate explicit physics simulation as a component.

Real-time physics: Currently physics-aware generation is slow, but optimization could enable real-time generation with physical accuracy.

The journey from teleporting basketballs to realistic bounces represents one of the most significant advances in AI video generation. Models have learned, if not to understand physics in the way humans do, at least to respect its constraints. For creators, this means fewer corrections, more possibilities, and videos that simply feel more real.