How Molmo 2's Open Release Changed an MIT Researcher's Approach to Fish Tracking

One salmon, one sonar video, and a research path reshaped by open weights

MIT researcher Kai Van Brunt set out to solve a very specific problem: automatically tracking salmon in sonar footage. Underwater acoustic video is nothing like standard camera footage — it's noisy, low-contrast, and fish outlines are often blurred. Traditional computer vision approaches struggle under these conditions, requiring extensive manual annotation and repeated tuning.

Molmo 2's open release gave him another path. Instead of training a purely vision-based tracking model, he built Molmo2Fish — an experimental model whose core capability isn't just "seeing" salmon, but also correcting tracking errors using everyday language. That shift in approach alone is worth unpacking.

What Molmo2Fish actually does

Using language as a correction interface

The key feature of Molmo2Fish is this: when tracking drifts off course, researchers can issue corrections in natural language rather than going back into an annotation tool to adjust frame by frame. This means frontline researchers — people who understand fish behavior but may not know how to fine-tune a model — can participate directly in the model's error-correction process.

Treating language as an interface for human-machine collaboration, rather than merely as training data, is an interesting pivot in multimodal model applications. It reduces the dependence of model iteration on engineering expertise, allowing domain knowledge to be injected into the system more directly.

Adapting the model to the sonar domain

There's a massive domain gap between sonar imagery and everyday photos. Molmo 2, as a general-purpose multimodal model, wasn't designed for underwater acoustic scenes. Van Brunt's work is essentially testing whether an open-weight general model, after moderate adaptation, can handle a highly vertical scientific task.

Molmo2Fish is still positioned as an experimental model, which means it may not yet reach production-level accuracy and robustness. But it demonstrates the feasibility of a path, not the completeness of a product.

What is Domain Gap

Domain gap refers to the mismatch between the distribution of training data and the distribution of target application data. Molmo 2 is primarily trained on internet-scale images — everyday photos, charts, documents — and the feature distribution its visual encoder learns differs greatly from the statistical properties of sonar imagery: sonar images are grayscale intensity maps formed by echo strength, lacking natural textures, shadows, and color cues, with target boundaries determined by abrupt changes in reflection intensity rather than geometric contours. This gap causes models to perform well on the source domain but degrade sharply on the target domain. Common mitigation strategies include: lightweight fine-tuning on the target domain, parameter-efficient adaptation with small amounts of labeled data (such as LoRA), normalization and augmentation on the input side, and using language prompts to constrain the task more explicitly. Van Brunt's work is essentially validating whether these techniques can pull a general model into an extremely vertical domain.

Why "open release" is the starting point here

Open weights let researchers "get their hands dirty"

If Molmo 2 were only a closed-source API, Van Brunt could have done little more than call the endpoint, tweak prompts, and accept the model's inherent performance on sonar images. Open weights change the researcher's scope of action: you can inspect the model's internal representations, fine-tune it for a specific domain, and embed it into your own data pipeline.

For scientific scenarios, this modifiability is often more important than absolute performance. The species involved in ecological monitoring vary enormously, imaging conditions differ across sites, and even the strongest general model needs domain adaptation.

The difference between open weights and open-source APIs

Open weights mean model parameters can be downloaded, inspected, modified, and redistributed, whereas a closed-source API only exposes an inference endpoint, letting users control only the input and a few hyperparameters. The implications for research differ significantly. With weights in hand, researchers can do several things an API cannot: first, inspect intermediate-layer representations to determine what the model has actually learned; second, continue training or fine-tune on their own data, encoding domain knowledge into the parameters; third, deploy the model in offline environments to handle sensitive or large-scale data; fourth, reproduce and audit results, which is critical to the credibility of scientific conclusions. The cost is that researchers must bear the compute, engineering, and maintenance burden themselves. So "open weights" is not a panacea, but it shifts the initiative in research from the service provider back to the user — which is precisely the precondition for interdisciplinary prototypes like Molmo2Fish to emerge quickly.

The real needs of ecological monitoring

Salmon population tracking is not an academic toy. Changes in salmon numbers directly affect fisheries management, habitat conservation, and ecological assessment. Relying on manual frame-by-frame interpretation of sonar video is costly, slow, and hard to keep consistent across different annotators. If automated tracking is reliable enough, it can significantly expand the spatiotemporal coverage of monitoring.

That's exactly where Molmo2Fish's value lies: it points toward a low-cost, replicable automated monitoring solution, not a custom tool for one particular stretch of river.

Why sonar is used for fish monitoring

Optical cameras underwater are limited by light attenuation, turbidity, and fish avoidance behavior, making long-term continuous observation difficult. Sonar actively emits sound waves and receives echoes, allowing it to work in darkness, turbid water, and even at night, with large coverage — well suited to long-term monitoring of population numbers and migratory behavior. The trade-off is low imaging resolution, no color information, and difficulty identifying species, often requiring indirect cues such as echo intensity, body size, and movement patterns. Traditionally, this interpretation relies on trained observers analyzing frame by frame, which is time-consuming and subjective. Precisely for this reason, sonar data is naturally suited to automated processing: large data volumes, high manual costs, and relatively formalizable interpretation standards — once a model is reliable enough, the benefits show up directly in the spatiotemporal coverage of monitoring.

What this means for AI applications

A general-purpose multimodal model going open source led to an interdisciplinary scientific application prototype within months — that chain of events itself is more informative than Molmo2Fish's current performance numbers.

It shows two things. First, the value of open weights is often realized in scenarios the publisher never anticipated — the model designers were thinking about general visual understanding, while the user took it to track salmon. Second, the "last mile" between multimodal models and vertical domains may not require a large engineering team; a single researcher with the right toolchain can bridge it.

For those building AI applications in ecology, oceanography, agriculture, and related fields, Molmo2Fish is more like a template: find a large, general-purpose open-source multimodal model, establish a human-machine correction loop using natural language, adapt it on your limited data, get it working first, then talk about accuracy.

One-line Summary

MIT researcher builds Molmo2Fish on open-weight Molmo 2, using natural language corrections for sonar salmon tracking.

Paragraph Summary

MIT researcher Kai Van Brunt set out to automatically track salmon in low-quality sonar footage, a task where traditional vision approaches fall short. Molmo 2's open release offered a new path. His Molmo2Fish doesn't just "see" salmon — it lets researchers issue corrections in everyday language, so domain experts who understand fish behavior but not model fine-tuning can participate directly in iteration. Open weights let researchers adapt the model to the highly vertical sonar domain rather than just calling a closed API. Though still experimental, Molmo2Fish validates a low-cost, replicable path for automated ecological monitoring, showing that open-source multimodal models often deliver value in scientific scenarios their creators never anticipated.