Florence-2-Large

Florence-2-Large is a vision foundation model that uses a sequence-to-sequence Transformer architecture, takes image and text prompt inputs, and outputs task results in text form.

Florence-2-Large was trained on the FLD-5B dataset, which covers 126 million images and contains 5.4 billion visual annotations.

Florence-2-Large supports multiple tasks such as image captioning, object detection, region captioning, visual grounding, and segmentation prompt generation, invoked by switching the task_input parameter, without the need for separate fine-tuning for each task.

Florence-2-Large's structured outputs such as coordinates and polygons need to be serialized into strings, must be parsed and converted before downstream use, and fault-tolerant handling must be prepared for format anomalies.

Florence-2-Large performs with limitations on small objects and fine-grained tasks (such as tiny defects in industrial quality inspection and small printed text), and may miss detections or produce vague descriptions.

Florence-2-Large's capabilities are limited to the prompt types trained during the FLD-5B annotation stage, and new vertical tasks that differ too much from the training data may produce hallucinations or generic descriptions.

Compared with florence-2-base, which has the same architecture, Florence-2-Large has more parameters and higher accuracy on complex scenes and fine-grained tasks, but the base version has 15%–20% faster inference and lower memory usage.

Compared with kosmos-2, the latter centers on vision-language alignment and is stronger at open-ended reasoning and spatial-relation question answering; Florence-2-Large is more efficient on batch annotation tasks and supports more structured output formats.

Compared with internvl3_5-30b, the latter has broader language understanding and reasoning capabilities but is slower and more compute-hungry; high-throughput annotation pipelines and single-task visual workloads are better suited to Florence-2-Large.

realistic-vision-v4.0 is an image generation model, opposite in function to Florence-2-Large and complementary to it.

The Florence-2-Large implementation on Replicate accepts three kinds of input—URI-format images, the task_input enum, and optional text_input—and its output contains text and an optional img (a visualization rendering bounding boxes or segmentation masks).

The task_input values are a fixed enum from the vocabulary of task prompts defined during model training (such as <CAPTION>, <OD>, <CAPTION_TO_PHRASE_GROUNDING>, etc.); choosing the wrong one or inventing a task name will not raise an error, and the model will merely degrade to generic description output.

The model files are in Hugging Face format and run on Replicate infrastructure via Cog.