Animation2Code: Evaluating Temporal Visual Reasoning in Video-to-Code Generation

Ji, Anya; Mudunuri, Abhijith Varma; Chan, David M.; Suhr, Alane

Animation2Code

Evaluating Temporal Visual Reasoning in Video-to-Code Generation

Anya Ji, Abhijith Varma Mudunuri, David M. Chan, Alane Suhr

University of California, Berkeley

NeurIPS 2026

Paper Code 🤗 Hugging Face Dataset Benchmark Dashboard

Animation2Code task overview: ground-truth code renders to animation videos, which models must de-render back into executable code. — **Animation2Code is the first benchmark to evaluate de-rendering of web animations into executable code.** Given a reference video, a model must produce self-contained HTML/CSS/JavaScript that, when executed, reproduces both the **appearance** and the **motion** of the original animation.

Abstract

While recent vision-language models (VLMs) have achieved significant improvements on static visual-to-code tasks such as generating code for webpages, charts, or SVGs, it remains unclear whether they can recover temporal dynamics when motion is present. To address this gap, we introduce Animation2Code, a benchmark for evaluating temporal visual reasoning via reconstructing executable web animation code from videos. Animation2Code consists of 1,069 web animation videos with diverse visual appearances and motion patterns, paired with corresponding HTML/CSS/JavaScript implementations. We propose two human-aligned metrics, appearance similarity and temporal similarity, which allow us to disentangle visual fidelity from temporal alignment when comparing animations rendered from generated code against ground-truth renders. Evaluating state-of-the-art VLMs on this dataset shows that current VLMs struggle to maintain temporal consistency in reconstruction, even when achieving high appearance similarity, including under fine-tuning and iterative refinement settings.

Evaluating De-Rendering

We evaluate generated animations along two complementary axes, both computed on the cropped animated region so they remain agnostic to absolute on-page positioning:

Appearance similarity compares DreamSim embeddings of frames aligned with Dynamic Time Warping, capturing color, shape, and style.
Temporal similarity compares motion trajectories (extracted with CoTracker3) via a direction- and speed-aware correlation, aggregated with Chamfer-style matching.

Appearance similarity via DreamSim + DTW, and temporal similarity via tracklet displacement correlation. — We isolate animated regions via cropping, compute **appearance similarity** using DreamSim with DTW alignment, and measure **temporal similarity** via tracklet displacement correlation aggregated with a Chamfer-style matching.

Results

Setting / Model	FPS	Exec (%)	Appearance	Temporal
Native video input
Gemini 3 Flash Preview	2	98.1	0.80	0.30
Gemini 3 Flash Preview	4	98.6	0.81	0.31
Gemini 3 Flash Preview	16	99.5	0.80	0.32
Qwen3-VL-8B-Instruct	2	85.5	0.67	0.23
Qwen3-VL-8B-Instruct	4	82.2	0.68	0.25
Qwen3-VL-8B-Instruct	24	84.6	0.69	0.24
Image frames input
GPT-5.4	2	100.0	0.84	0.29
Gemini 3 Flash Preview	2	100.0	0.80	0.30
Claude Sonnet 4.6	2	99.5	0.82	0.29
LLaMA 4 Scout	2	97.7	0.62	0.21
Qwen3-VL-8B-Instruct	2	80.4	0.70	0.24
SFT / refinement (Qwen3-VL-8B-Instruct, video input)
+ LoRA	2	86.4	0.59	0.22
+ Full SFT	2	87.4	0.56	0.21
+ Iterative Refinement	2	85.5	0.73	0.28

For Gemini and Qwen3-VL, we report three input FPS settings: 2 FPS and the best input FPS for each metric based on a full FPS sweep from 0.5 to 24 FPS. Exec (%) is the share of examples whose generated code runs successfully. Best score per metric in bold blue.

Takeaway. Execution is nearly saturated and appearance is strong, but temporal similarity remains uniformly low across every model and training setting. High appearance similarity does not translate into faithful motion reconstruction. Richer temporal signal (native video, higher FPS) yields only marginal gains.

Qualitative Examples

Ground Truth

GPT-5.4

Gemini-3

Claude

Qwen3-VL

LLaMA 4

All models struggle with precise motion and spatial relationships, even when overall appearance is partially correct. Each row shows one example: the ground-truth animation (blue) alongside the reconstructions rendered from each model’s generated code.

Human-Aligned Evaluation

We collected 600 pairwise preference comparisons from 65 Prolific annotators across overall quality, appearance, and temporal similarity. Inter-annotator agreement is strong for overall and appearance (Krippendorff's α = 0.81) and lower but reliable for temporal (0.73), as motion is inherently harder to judge. Appearance and temporal similarity capture complementary aspects of animation quality. Combining both metrics with logistic regression (Joint LR) achieves best agreement with human annotations across all dimensions, without relying on VLM-based judging.

Inter-annotator agreement and ROC-AUC of metrics against human preference. — (A) Inter-annotator agreement per judgment dimension. (B) ROC-AUC for predicting human preference from metric deltas, two VLM-based baselines (VLM Judge and WebVR-IM), and a joint logistic regression.

BibTeX

@article{ji2026animation2code,
  title         = {Animation2Code: Evaluating Temporal Visual Reasoning in Video-to-Code Generation},
  author        = {Ji, Anya and Mudunuri, Abhijith Varma and Chan, David M. and Suhr, Alane},
  year          = {2026},
  eprint        = {2606.28593},
  archivePrefix = {arXiv},
  url           = {https://arxiv.org/abs/2606.28593},
}