Can LLMs Run a Film Crew? A Benchmark for Creative Control

When you're making a short film with AI video tools, what usually breaks everything isn't that any single shot looks bad — it's continuity falling apart. You lock in a character, a location, a prop, and a few steps later the model quietly forgets one of them.

A developer who works on AI video tools full-time designed a creative benchmark for a Kaggle challenge that asks not "which LLM writes the best story," but the question an actual filmmaker cares about:

When a director keeps changing their mind, can an LLM maintain creative control — from concept to shot list — across the entire film?

Test Design: One Model Playing the Entire Crew

Each model receives a one-sentence story premise and a handful of hard rules, then plays the entire crew within a single conversation:

1. Screenwriter: Write a three-act story outline (exactly N characters, a limited number of locations, a banned-words list, and one required story element)
2. Casting Director: One entry per character
3. Production Designer: Sets, props, and color palette
4. Director of Photography: A shot list where every shot references specific sets, characters, and props

The director then issues three revision notes in sequence — without explaining their downstream consequences:

• Period shift: "Move the whole story to 1890s London." Every phone, laptop, and fluorescent light must vanish from everywhere.
• Budget cut: "We're cutting Set X." Every scene set there must move, and the final count must be one set fewer.
• Character merge: "Combine characters A and B into a new character C." Neither old name can appear anywhere.

The key mechanic: after each note, the model returns only the affected shots — just like a real revision request. Untouched shots stay as-is. That means the model has to figure out which shots are affected on its own. Miss one, and an old prop or old character name is still sitting in the shot list.

The full benchmark covers 12 film concepts (a pharmacy noir, a lighthouse^Ad keeper's log from the future, a retired museum guard planning a heist, and more), split across two Kaggle tasks — Creative Control A and B, six concepts each — with the final score averaging both halves.

Scoring: Mostly Code, Minimal AI Judging

The scoring system deliberately minimizes subjectivity. Constraint compliance (cast size, set count, banned words, required elements), consistency (every character, set, and prop in the shot list must actually exist in the casting and production design documents), and execution of the director's notes are all checked by code. Only "story quality" goes to a judge model, which answers five yes/no questions.

A revision that's only half-executed counts as not executed. As the author puts it: "A laptop in 1890s Vienna isn't a 5% problem — it's a reshoot."

The three-act structure is Hollywood's foundational scriptwriting framework: Act One establishes the world and characters and triggers the conflict; Act Two develops the tension and drives toward a climax; Act Three resolves the conflict and closes the story. Screenwriting theorist Syd Field systematized the framework in 1979, and it has been an industry standard ever since. Using it as a benchmark constraint means the model isn't just free-generating a story — it's creating within a structure with explicit requirements, which is closer to a real professional production environment and makes the outputs meaningfully comparable.

Results: Flagship Pricing Doesn't Guarantee Creative Control

The author selected a large and a small model from several labs, plus an open-source model, to test the core question: does paying more for a flagship model actually buy you creative control?

Model	Overall	A / B	Director Notes	Story Quality
GPT-5.5	0.94	0.96 / 0.92	0.94	0.85
Gemini 3.7 Flash	0.90	0.89 / 0.90	0.85	0.73
DeepSeekAd R1	0.88	0.87 / 0.90	0.83	0.70
Gemma 4 31B	0.84	0.85 / 0.82	0.71	0.65
Claude Sonnet 5	0.80	0.91 / 0.69	0.75	0.82
GPT-5.4 nano	0.76	0.74 / 0.77	0.40	0.70
Claude Haiku 4.5	0.69	0.64 / 0.73	0.50	0.62
GLM-5	0.64	0.55 / 0.74	0.75	0.69
Claude Opus 5	Half (A only)	0.43 / –	0.67	1.00

Notably, Claude Opus 5 achieved perfect story quality scores on every film it completed — but its tendency to think too long meant reasoning plus output frequently hit the 16,000-token limit, causing it to time out and lose points on the period-shift task, which required rewriting every shot. At roughly $19 per full run, it also exceeded Kaggle's $10-per-day per-model budget, so it only completed half the test.

Scores across the two halves were highly consistent for most models (within 0.03), indicating the scores reflect the model itself rather than luck of the draw on which prompts it got.

Six Key Findings: Each Model Breaks in Its Own Way

Following Instructions Separates Models More Than Writing Does

Excluding the three flagships, every model's story quality score lands between 0.62 and 0.73. But director-note scores range from 0.40 all the way to 0.94. In other words, if you only read the stories, the models look roughly the same. Hand them a revision memo, and the gaps appear immediately.

Every Model Has Its Own Failure Mode

• GPT-5.4 nano forgot the world had changed: In 7 of 12 films, modern objects survived the "move to the 1890s" — laptops and phones in 1890s Vienna, GPS in 1890s Norway, a refrigerator in 1890s Ohio.
• Gemini 3.7 Flash was the most consistent model on the board, and its only recurring "failure" later turned out to be the author's own bug (see below).
• Gemma 4 handled period shifts cleanly but botched cast merges: It never left modern objects behind, but in 4 films it reduced the wrong cast size during merges.
• Claude Haiku 4.5 only updated shots it touched, forgetting the rest: After merging characters, old names remained in shots it hadn't revised — in 5 of 11 films.
• GPT-5.5 knew when to rewrite everything: On the period shift it rewrote every single shot (the safest approach when the whole world changes), while on budget cuts and merges it only touched what needed changing — and got it right. That's exactly what a script supervisor does, and it's why GPT-5.5 led the field.

On the "is flagship worth it" question, the answer depends on the lab. GPT-5.5 over GPT-5.4 nano is the largest jump on the board (director notes: 0.94 vs. 0.40), but the cheaper Gemini 3.7 Flash and the open-source DeepSeek R1 both outperformed the two Claude flagships on instruction-following.

When Rules Conflict, Models Pick a Side

The author openly admits a design error: the opening rules said "number of sets between 2 and N," but the budget-cut note said "cut one set." For any film that started with exactly two sets, these two instructions can't both be satisfied — and the models split into clear camps.

Gemini's response to "cut the restaurant" was to build a head chef's office; "cut the lighthouse room" got a storm porch added instead. The original rule won, and the director saved nothing. Haiku and DeepSeek always sided with the most recent instruction. Most flagships had designed three sets from the start, neatly sidestepping the conflict.

The author's scoring script originally only accepted "reduce to one set," which unfairly penalized models that upheld the original rules. After rescoring to accept either choice as correct, Gemini moved from 0.90 to 0.93, Gemma from 0.84 to 0.87, and all other changes were within 0.01 — rankings were unchanged.

Everyone Has the Same Writing Weakness

With the exception of Claude Sonnet 5, the judge model's most common complaint was strikingly consistent: the criterion "each character wants something different, and those desires create conflict" went unmet in 6 to 9 films per model. LLM-generated shorts tend to have one character with a goal while everyone else exists purely to react. Worth keeping in mind if you're using these models for story development.

Cast Without Shooting

A recurring consistency error: a character gets cast and described, but never appears in a single shot. The model hired them and then forgot to put them on screen.

The Second Task: Do Models Think Like Cinematographers?

The second task asked models to provide cinematography direction for eight dramatic moments, with each moment framed three ways:

• Explicit: "A woman realizes everyone at the dinner party knows her secret. Use a dolly zoom." — Does it know the technique?
• Intent-based: "…Make the audience feel sudden isolation in a crowded room." — Can it translate a feeling into a shot choice?
• Implied: Just the moment itself. — Will it reach for cinematic language on its own?

The results show that cinematography vocabulary is a solved problem: every model planned shots that were filmable, specific, and motivated, with total scores clustered between 0.91 and 0.96. The author ran the same prompts twice, and most metrics fluctuated by 0.1–0.3 between runs, so he declined to rank models on this task.

Two findings held stable across both runs:

• Left to their own devices, models default to the most clichéd choices: When told which technique to use, 93% of plans avoided clichés. Given only the moment, that dropped to 75% (first run) and 82% (second run).
• Knowing the name of a technique isn't the same as understanding how it works: Claude Haiku described a split diopter as keeping one actor in focus while blurring another — the exact opposite of what it actually does, which is keeping both near and far subjects in focus simultaneously. When asked for "one continuous uninterrupted Steadicam shot," both Haiku and DeepSeek R1 planned four separate shots with cuts in between.

The dolly zoom (also known as the "Hitchcock zoom" or "Vertigo effect") combines opposing camera and lens movements: the camera physically moves toward (or away from) a subject while the focal length changes in the opposite direction, keeping the subject the same size in frame while dramatically compressing or stretching the background — producing a strong sense of spatial distortion and psychological impact. Hitchcock used it extensively in Vertigo (1958); Spielberg later made it iconic in Jaws. The split diopter is a half-circle close-up filter mounted in front of the lens that allows objects at two very different distances to appear in sharp focus simultaneously — the optical opposite of depth-of-field control, essentially a trick that "cheats" the eye. Brian De Palma is its most recognizable practitioner. Both are techniques with precise physical mechanics, so whether a model describes them accurately reveals the depth of its cinematography knowledge — not just vocabulary recall.

How to Read This Benchmark

The author honestly lists several caveats: 12 film concepts is a small sample; one missed revision note shifts a score by roughly 2%, so close scores should be treated as ties; the judge model is Gemini 3.8 Flash, which may favor Gemini-family stories, so the story quality column deserves extra skepticism — though director notes and consistency are checked by code and are unaffected.

Kaggle's official leaderboard averages all three tasks. Because the cinematography task pushes every model near the ceiling, it compresses the spread — but the order at the top is consistent: GPT-5.5 (0.94), Gemini 3.7 Flash (0.92), DeepSeek R1 (0.90).

The most valuable thing this benchmark offers may not be the rankings themselves, but the ability to quantify a long-overlooked capability: maintaining long-range consistency through repeated revisions is the real threshold for AI entering a creative workflow. Any model can generate a pretty shot on demand. But when the director changes their mind for the third time, remembering exactly which props need to disappear and which old character names need to be erased — that's the dividing line between a toy and a production-ready tool.

Kaggle is Google's data science competition platform, known for hosting machine learning challenges. It provides participants with limited compute and API quotas; the "$10-per-day per-model budget" mentioned in this article is the API spend cap Kaggle set for this benchmark. That constraint had real consequences during evaluation: Claude Opus 5, at roughly $19 per full run, far exceeded the daily limit and could only complete half the test. This also illustrates a constraint just as real in production environments: the model that performs best with an unlimited budget isn't necessarily the most cost-efficient choice.