The THPL Framework: Driving Precision Feeding Decisions for Rainbow Trout with Vision-Language Models

When Aquaculture Meets Large Language Models

In Recirculating Aquaculture Systems (RAS), precision feeding is central to reducing costs and improving fish welfare. Overfeeding pollutes water quality and wastes feed, while underfeeding stunts fish growth. Yet existing methods share a critical gap: there is a lack of "cognitive alignment" between machine recognition of fish behavior and human aquaculture expertise, making it difficult to translate observations into actionable, explainable feeding decisions.

A new paper on arXiv proposes a generative feeding decision framework called THPL, designed specifically for rainbow trout (Oncorhynchus mykiss) in recirculating aquaculture systems. It attempts to combine continuous fish school kinematic information with the reasoning capabilities of large language models (LLMs), offering a novel decision-support paradigm for precision aquaculture.

THPL's Three-Stage Technical Pipeline

The framework's core idea is to decompose the process from "observing fish" to "making decisions" into three progressive steps, abstracting visual information layer by layer into evidence that a language model can understand.

Step 1: From Trajectories to Activity Coefficient

Researchers first use a module called Fishsort to extract fish school movement trajectories, then construct an Activity Coefficient (AC) to quantify feeding intensity. The significance of this metric is that it transforms the "activeness of fish competing for food" — something difficult to judge consistently with the naked eye — into a computable continuous value.

The validation results are impressive: AC shows a statistically significant monotonic positive correlation with expert-annotated feeding intensity, achieving a Spearman correlation coefficient of ρ = 0.925, p < 0.001. In other words, this automatically computed coefficient aligns closely with the subjective judgments of experienced aquaculture specialists, demonstrating that it genuinely captures the essential characteristics of feeding behavior.

The Spearman correlation coefficient (ρ) is a nonparametric statistical method for measuring the strength of a monotonic relationship between two variables, ranging from -1 to 1, where values closer to ±1 indicate stronger correlation. Unlike the Pearson correlation coefficient, it does not require normally distributed data or assume a linear relationship between variables. This makes it particularly suitable for evaluating the consistency between an "automatically computed metric" and "human subjective ratings" — the latter being inherently ordinal, non-linear scale data. ρ = 0.925 with p < 0.001 means that statistically, the ranking of AC values almost perfectly matches the ranking of expert feeding intensity scores, and this consistency is far from coincidental (the extremely small p-value indicates a very low false-positive probability). For an automated metric intended to replace human judgment, this result is an important endorsement of methodological credibility.

Step 2: Hierarchical Behavior Encoder

Next, the Hierarchical Behavior Encoder (HBE) models two levels separately using a Temporal Transformer and a Set Transformer — the temporal evolution of individual fish, and the collective dynamics of the school. It transforms raw trajectory tensors into a "dual-evidence" representation: explicit physical tokens and implicit soft tokens.

This dual-evidence design is the technical core of the entire framework. Explicit physical tokens provide interpretable kinematic grounds, while implicit soft tokens capture collective behavioral patterns that are difficult to describe with fixed rules. The two complement each other, giving the downstream language model a solid "physical grounding."

The Temporal Transformer and Set Transformer are two attention mechanism variants tailored for different data structures. The Temporal Transformer excels at processing time-series data, capturing dependencies along the temporal dimension of trajectories through self-attention — for example, changes in a single fish's speed and direction across consecutive frames. The Set Transformer is designed for unordered sets, where input order does not affect the output, making it suitable for modeling the overall state of the fish school — the order of individuals within the school is arbitrary; what matters is their collective distribution characteristics. The combination of the two enables HBE to simultaneously capture "one fish's behavioral evolution over time" and "the spatial distribution of the entire school at a given moment," forming a dual-layer perception from microscopic individual to macroscopic group — precisely the two complementary dimensions needed to understand feeding behavior.

Step 3: LLM Fine-Tuning and Counterfactual Preference Optimization

In the final step, these behavior tokens are integrated with environmental parameters, metadata, and expert rules, and a large language model is fine-tuned via LoRA. Building on this, the researchers further introduce counterfactual multimodal Direct Preference Optimization (counterfactual mDPO) to strengthen the model's causal reasoning and prevent it from merely learning to apply templates.

LoRA (Low-Rank Adaptation) is a parameter-efficient method for fine-tuning large language models. Its core idea is to avoid updating all model parameters; instead, low-rank decomposition matrices are injected alongside the original weight matrices, and only these small additional parameter blocks are trained. This allows the model to be adapted to a specific vertical domain task at minimal computational cost while preserving the general capabilities of the pre-trained model. In THPL, LoRA enables researchers to professionally adapt a large language model using a relatively limited dataset of rainbow trout aquaculture data, without training from scratch or fully fine-tuning billions of parameters. Counterfactual multimodal DPO (Direct Preference Optimization) is a further alignment step on top of LoRA: it trains the model to learn human preferences by constructing contrastive sample pairs of "good decisions vs. bad decisions," and reinforces the model's causal judgment through counterfactual reasoning (i.e., "if conditions change, how should the decision change?"), preventing it from applying fixed answer templates based on surface-level patterns alone.

Experimental Results: Dual-Evidence Tokens Are the Key

Ablation experiments reveal the contribution of each module, with the most compelling data being the leap in decision accuracy.

A text-only baseline model achieves a decision accuracy of only 33.33%; once dual-evidence tokens are introduced, accuracy shoots up to 93.33%. This enormous gap powerfully validates a key insight: continuous spatiotemporal tokens provide the essential physical grounding for large language models — textual descriptions alone fall far short.

Building on this, counterfactual mDPO delivers further across-the-board improvements over standard LoRA:

• Decision accuracy rises from 93.33% to 96.67%
• METEOR (a text generation quality metric) improves from 58.10% to 85.30%
• Self-BLEU-2 drops from 58.79% to 52.88% (a lower value means more diverse, less repetitive outputs)
• Distinct-3 improves from 6.68% to 7.81%

This set of results demonstrates that mDPO effectively suppresses the model's tendency toward templated responses and execution bias, while reinforcing causal consistency and operational safety. In real aquaculture scenarios, "operational safety" and "decision interpretability" are often more important than raw accuracy alone.

What This Means for Precision Aquaculture

THPL's value lies not only in metric improvements, but also in demonstrating a technical pathway that integrates visual perception, kinematic modeling, and language model reasoning into a unified whole. Traditional feeding systems typically rely on fixed schedules or simple threshold triggers, whereas THPL enables the system to "understand" the current state of the fish school and provide feeding recommendations with causal explanations.

From a broader perspective, this research exemplifies a typical pattern for deploying multimodal large models in vertical industries: first, transform domain-specific expert experience that is difficult to quantify into computable metrics (such as AC); then, use structured encoders to bridge perception and language; and finally, use preference optimization to align model outputs with industry norms. This methodology is equally applicable to other agricultural and industrial scenarios that require a closed loop of "behavior recognition — expert knowledge — actionable decisions."

Of course, as a newly published study, THPL's validation scale and real-world deployment status remain to be seen. The improvements in decision accuracy were achieved under relatively controlled experimental conditions; whether they hold up robustly in complex, variable real-world aquaculture environments will require larger-scale, long-term validation. Nevertheless, the paradigm it proposes — "continuous kinematics + LLM reasoning" — genuinely opens up a new horizon of possibility for precision aquaculture.