A Beginner's Guide to Vision Foundation Models: From Pre-Training to Downstream Tasks

Vision Foundation Models have become one of the hottest research directions in the computer vision community. This article is based on a systematic course on vision foundation models, outlining the knowledge framework, technical evolution, and learning path of this field to help readers build a comprehensive understanding.

Two Core Knowledge Areas in Vision Foundation Models

The knowledge landscape of vision foundation models can be cleanly divided into two parts: pre-training vision foundation models and adapting them to downstream tasks.

Pre-training refers to the process of training a large model from scratch. Whether it's GPT, ERNIE Bot, or the various foundation models released by major tech companies, they all fall under this pre-training paradigm. Downstream task adaptation, on the other hand, is about taking an already-trained foundation model and applying it effectively to specific business tasks.

The course follows this same structure: Chapter 1 covers self-supervised learning (the foundation of vision foundation models), Chapter 2 focuses on vision-only foundation models, and Chapter 3 covers multimodal vision foundation models — all three belong to the pre-training section. Starting from Chapter 4, the content shifts to downstream task transfer. For core network architectures or modules, the course also includes code demonstrations to help build intuition for the underlying principles.

Can University Labs Actually Work on Vision Foundation Models?

A common question is: with limited compute resources in most labs, can ordinary researchers realistically do foundation model pre-training? The answer is more encouraging than you might expect.

NLP foundation models are indeed enormous, and most labs don't have the compute to train them. But vision foundation models in the CV domain often don't have nearly as many parameters. Even if training a general-purpose model for the entire CV field is out of reach, focusing pre-training on a specific domain or dataset means a few RTX 3090 GPUs can sometimes be enough to train a domain-specific visual foundation model.

And even if GPU resources are tight, there's no need to be discouraged — pre-training itself is a hot research topic. Using a small ViT model and demonstrating results that outperform other models of similar scale is enough to be publishable. This creates a realistic entry point for researchers with limited resources.

Prerequisites for Learning

The course tries to keep the barrier to entry as low as possible, but some foundational knowledge is still required:

• Deep learning basics: Familiarity with architectures like CNN and Transformer, and an understanding of the training and inference workflow;
• PyTorch: At least some hands-on experience developing with PyTorch;
• CV task awareness: A basic understanding of what common computer vision tasks like classification, detection, and segmentation involve;
• A genuine interest in computer vision and large models.

If any of these foundations are shaky, it's recommended to fill the gaps through courses or online blog posts — a modest amount of study is usually enough to get up to speed.

The Evolution of Large Models

The rise of large models was largely driven by breakthroughs in NLP. Back in October 2018, BERT — a Transformer-based self-supervised learning approach — suddenly achieved dramatic performance gains. Researchers realized that simply scaling up model size and data volume yielded significantly better performance. This phenomenon became known as emergent intelligence.

GPT-3 followed, reaching hundreds of billions of parameters and demonstrating few-shot and zero-shot capabilities. In other words, once a large model is trained, it can adapt to a wide variety of tasks with minimal examples — AI began to acquire the human-like ability to learn new things from just a few demonstrations.

Multimodal approaches then emerged, such as the Wudao series from Tsinghua University and BAAI, which incorporated image, audio, and other modalities on top of NLP foundations. There were also foundation models trained purely on visual data, such as the general-purpose vision foundation model released by the Shanghai AI Laboratory. The explosive mainstream adoption of ChatGPT brought large models into the public spotlight, and GPT-4V (Vision), as a multimodal model, can now simultaneously process and generate both text and image content.

Emergent Abilities refer to capabilities that suddenly appear in a model once its parameter count or training data volume exceeds a certain threshold — capabilities that are essentially nonexistent at smaller scales. This is not a linear improvement but rather a phase-transition-like qualitative leap. Researchers have found that as model scale grows from hundreds of millions to hundreds of billions of parameters, abilities such as arithmetic reasoning, logical analogy, and multi-step problem solving "emerge" — while being nearly absent in smaller models. GPT-3's few-shot and zero-shot capabilities at 175 billion parameters are a prime example: without any gradient updates for new tasks, the model can complete them based solely on a handful of demonstrations, closely mirroring how humans generalize from limited examples. The discovery of emergent abilities fundamentally changed the direction of AI research — shifting focus away from designing task-specific architectures and toward a unified paradigm of "bigger models + more data."

Three Categories of Large Models

Based on this evolution, current large models can be roughly grouped into three categories:

• Large Language Models (LLMs): Text-only processing;
• Vision-only foundation models: Trained exclusively on visual data;
• Multimodal large models (VLMs, Vision Language Models): Most commonly a combination of language and vision, but can also encompass audio, autonomous driving sensor data, specialized medical imaging, and other modalities.

Any model that goes beyond a single modality falls into the multimodal category.

VLM (Vision Language Model) technology centers on aligning a visual encoder with a language model. Representative work includes OpenAI's CLIP, which uses contrastive learning to jointly train an image encoder and a text encoder on 400 million image-text pairs, placing images and their corresponding descriptions closer together in the same vector space. This alignment enables zero-shot image classification without any additional annotation. Subsequent models such as DALL-E, Flamingo, and LLaVA further integrate language generation with visual understanding, enabling capabilities like visual question answering, image generation, and multi-turn visual dialogue. Compared to vision-only foundation models, VLMs have the advantage of allowing natural language to describe task objectives, significantly reducing downstream adaptation costs. The trade-off is that training requires large amounts of high-quality image-text paired data, and language biases can affect the accuracy of visual reasoning.

From Generalist to Specialist: The Ideal Vision Foundation Model

Drawing on a framework from a lecture by Professor Dai Jifeng of Tsinghua University, the ideal vision foundation model would take as input massive image data, paired image-text data, and pure text data. The goal is to train a large-scale general visual perception model — essentially a backbone network for the vision domain — capable of solving hundreds of visual tasks.

Training methods can include self-supervised, weakly supervised, or fully supervised learning. Given enough data and a large enough model, a universal visual representation can be obtained.

But training the backbone is not the end goal. The CV foundation model must still be transferred to various downstream tasks for real-world deployment — a process known as knowledge transfer or transfer learning. Since downstream task data often differs from pre-training data, sometimes with no overlap at all, the core challenge of knowledge transfer is bridging the Domain Gap between them.

Here, the foundation model is called the generalist model, while the task-specific model for a particular problem is called the specialist model. The entire process is one of going from generalist to specialist. This closely mirrors the human education system: the generalist model is like compulsory primary and secondary schooling, building broad foundational knowledge across many domains without deep specialization; the specialist model is like a university major, focusing deeply on a specific field.

Foundation Models: A Paradigm Shift

Before the term "large model" became popular, academia generally used the term Foundation Model — both refer to the same thing.

A foundation model is defined as a model trained on large-scale data, most commonly through self-supervised or semi-supervised learning (though fully supervised approaches exist as well), and capable of being adapted to a wide range of downstream tasks.

The entire AI field has transitioned from traditional training paradigms to the Pre-Training + Fine-Tuning era of foundation models. We no longer need to collect separate datasets and train individual models for each task, nor do we need large amounts of manual annotation. Instead, unsupervised learning on massive data produces a general model, which can then be transferred to a wide range of downstream tasks at low cost with minimal labeled samples.

Foundation models can also be hierarchical: starting from a broad CV foundation model, transferring to a medical imaging foundation model, and then further specializing into models for pathology images, MRI, CT scans, and so on — achieving varying degrees of specialization.

The core value of the Pre-Training + Fine-Tuning paradigm lies in decoupling the two stages of "learning general representations" and "learning task-specific behavior." The pre-training stage uses massive unlabeled or weakly labeled data to help the model acquire rich world knowledge and feature representations. The fine-tuning stage then performs lightweight updates on a small amount of labeled downstream task data, enabling rapid adaptation to specific scenarios. Compared to training from scratch, fine-tuning typically only updates a subset of parameters (e.g., the final few layers), and techniques like Prompt Tuning, Adapters, and LoRA have emerged as parameter-efficient fine-tuning (PEFT) methods that reduce computational costs even further. Domain Gap refers to the distributional mismatch between pre-training data and downstream task data — for example, a model pre-trained on natural images may perform dramatically worse when directly applied to medical pathology images. Bridging this gap through transfer learning techniques is the central challenge in deploying vision foundation models in practice, and forms the main focus of the downstream task adaptation section of the course.

Conclusion

Vision foundation models are one of the most active and rapidly evolving directions in today's CV research community. Understanding the dual-track framework of "pre-training + downstream adaptation" and grasping the transfer path from generalist to specialist are the keys to entering this field. In deeper study ahead, self-supervised learning, vision-only foundation models, multimodal models, and various downstream transfer methods will all unfold around this central theme.