TIDE: Distilling Text-to-Video Prompt Optimization into Reusable Skills

Text-to-video (T2V) generative models are extremely sensitive to prompt precision—the same idea, phrased differently, can produce wildly different videos. To address this, researchers typically adopt a "generate-evaluate-regenerate" iterative search strategy: repeatedly calling the T2V model to generate videos, then using a vision-language model (VLM) to score them, until a satisfactory result is found. While effective, this approach is extremely expensive, and each optimization is equivalent to an independent search process that cannot accumulate experience.

Recently, a framework called TIDE (Trajectory-Informed Skill Distillation) proposed a different approach: distilling this time-consuming search process into an interpretable, reusable prompt rewriting skill. The work has been open-sourced on GitHub (github.com/xiningin/TIDE_T2V).

Core Idea: From "Searching Every Time" to "Learning to Rewrite"

The core insight of TIDE is that prompt optimization is essentially a set of rules that can be learned and reused, rather than a search starting from scratch each time.

Specifically, TIDE models prompt rewriting as a set of source-conditioned rule modules, each targeting an independent semantic dimension—such as subject description, action details, scene lighting, camera movement, etc. This modular design makes the entire rewriting process interpretable: users can see which dimensions were adjusted and how.

The key difference is that TIDE does not require visual search or backbone model fine-tuning during inference. Once the skill is distilled, it activates relevant rules in a single LLM call, simultaneously generating positive and negative prompts, and then performs only one T2V generation. The entire process has no test-time visual feedback loop.

"Source-conditioned" in source-conditioned rule modules means that a rule module's activation is conditioned on the "source prompt" (i.e., the user's original input prompt). For example, a module responsible for the "camera movement" dimension is only activated and produces rewriting suggestions when the source prompt involves or implies some kind of camera movement need, rather than indiscriminately applying all rules to every prompt. This design draws on the idea of conditional routing in Mixture of Experts (MoE), allowing the system to maintain modular interpretability while avoiding interference between rules of irrelevant dimensions. Each module is essentially a structured natural language instruction or several rewriting paradigms, based on which the LLM performs targeted modifications to the source prompt during inference, so the entire rewriting process can be both read by humans and programmatically composed.

Training Efficiency: 100 Prompts, 14 Hours

TIDE's training cost is far lower than comparable methods. According to the paper, the entire evolution process uses only 100 prompts and takes 14 hours.

This figure is noteworthy. Traditional search-based prompt optimization methods require repeatedly generating and evaluating videos during inference, with computational overhead growing linearly with optimization rounds. TIDE front-loads this cost into a one-time distillation, with nearly zero additional overhead at inference. The paper claims that the distilled TIDE achieves performance comparable to computationally expensive search methods in prompt optimization.

In other words, TIDE trades a one-time investment for sustained efficiency advantages—this "train once, benefit forever at inference" model aligns with the current trend in the large model field evolving from prompt engineering toward skill distillation.

Skill distillation is the core paradigm of this paper, referring to compressing some tacit capability that relies on extensive trial and error (such as "how to rewrite prompts to make the video model output better results") into an explicit, directly callable rule asset. Its difference from knowledge distillation lies in the fact that the latter typically transfers the capabilities of a large model to a small model, whereas skill distillation transfers "the method of doing things" rather than "the model's parameters." In the context of T2V prompt optimization, what is distilled is the rewriting patterns precipitated from search trajectories—the original search process produces a large number of (original prompt → rewritten prompt → score) samples, and distillation extracts stable, cross-sample reusable rewriting rules from them. The "train once, benefit forever at inference" model is therefore particularly attractive for businesses that frequently call T2V models.

Cross-Model Transfer: Performance on Unseen Models

A further detail about TIDE's transfer capability.

When TIDE is transferred to video generation models it has not seen before, it surpasses methods specifically designed or trained for those models. This is quite illuminating: what TIDE learns may not be the preferences of a specific model, but more general knowledge of "what a good prompt should look like."

This means prompt rewriting skills have a certain degree of model-agnosticism. For application scenarios that require frequent switching of T2V models (e.g., due to cost, availability, or quality considerations), this transferability has practical value—there is no need to retrain a set of optimization strategies for each model.

Model-agnosticism here means that the rewriting rules TIDE learns do not encode the parameter preferences of a specific T2V model, but instead correspond to common requirements of general video generation tasks, such as clear subjects, explicit actions, and controllable lighting and camera. This forms a contrast with the common dilemma of prompt engineering: prompt templates finely tuned for model A often fail on model B, because different models have different sensitivities to sentence patterns, keyword weights, and description granularity. TIDE's ability to surpass methods dedicated to the target model suggests that its rules capture something closer to a "universal grammar of visual language" rather than the quirks of a specific model. This also explains why it is suitable for application scenarios requiring frequent model switching.

Industry Perspective: Prompt Optimization Is Moving Toward "Skillization"

TIDE reflects a broader trend: the AI community is shifting from manual prompt engineering to systematic skill distillation.

Manual prompt tuning relies on personal experience and repeated trial and error, making it difficult to scale and reuse. Methods like TIDE attempt to make tacit "prompt techniques" explicit as rule modules, making them interpretable, transferable, and reusable. For content creators and AI application developers, this means that in the future they may no longer need to separately explore prompt writing methods for each model, but instead call distilled universal rewriting skills.

Of course, TIDE currently only targets T2V scenarios, and whether its rule modules can be extended to image generation, 3D generation, or other multimodal tasks remains to be verified. Although 14 hours of training time is not long, for rapidly iterating application scenarios, how to further compress distillation costs is also a direction worth watching.

The release of open-source code provides a foundation for follow-up research. The pain point of T2V models being sensitive to prompts will not disappear in the short term, and the idea of turning the optimization process into an asset may be a direction with more long-term value than "better prompt templates."

Negative prompts are a common control mechanism in T2V and image generation models, used to explicitly tell the model "what not to generate," such as blur, distortion, extra limbs, watermarks, etc. They work together with positive prompts to jointly constrain the generation space. The significance of TIDE generating both positive and negative prompts lies in the fact that many image quality and structural issues are difficult to eliminate by "adding positive descriptions," whereas "explicitly stating what to avoid" is often more efficient. Having rule modules output both types of prompts is equivalent to expanding prompt optimization from "enhancing what is wanted" to "simultaneously suppressing what is not wanted," providing more complete coverage and aligning more closely with the actual practices of experienced users manually tuning prompts.

One-line Summary

TIDE distills iterative text-to-video prompt optimization into reusable rule modules, requiring only one generation at inference and enabling cross-model transfer.

Paragraph Summary

TIDE (Trajectory-Informed Skill Distillation) proposes distilling the expensive process of text-to-video prompt optimization into an interpretable, reusable rewriting skill. Traditional methods rely on iterative "generate-evaluate-regenerate" search, requiring repeated calls to the T2V model and VLM scoring, which is costly and cannot accumulate experience. TIDE instead models rewriting as a set of source-conditioned rule modules, each targeting independent semantic dimensions such as subject, action, lighting, and camera. At inference, only one LLM call activates relevant rules and simultaneously generates positive and negative prompts, without visual search or backbone fine-tuning. Its training uses only 100 prompts and takes 14 hours, yet performs comparably to expensive search methods; when transferred to unseen video models, it even surpasses methods specifically designed for them, demonstrating model-agnosticism. This work reflects the trend of prompt optimization moving from manual engineering toward skill distillation.