Multimodal Large Language Model

Multimodal large models convert images into high-dimensional vector representations through a Vision Encoder, which are then fused with the text comprehension capabilities of the language model.

Current large models processing video do not understand it frame by frame; instead, they use intelligent sampling to extract key frames from the video (typically 1–4 frames per second) as image inputs for analysis.

MiMo-V2.6 covers multiple modalities including text, image, and speech, making it an all-modality model.

The architectural core of a multimodal large model must include a large language model (LLM) as its backbone.

If the input is audio, the Vision Encoder can be replaced with a Speech Encoder to encode and align the data before passing it to the large language model.

Multimodal large language models (MLLMs) are large-scale neural network models capable of simultaneously processing and understanding multiple modalities such as text, images, audio, and video, typically based on the Transformer architecture, and acquiring cross-modal alignment capabilities through pretraining on large-scale multimodal data.

Among mainstream large models on the market, there are almost none that meet the requirement of native real-time multimodal input and output (input supporting speech, images, text and video, and output also multimodal including speech).

In the WOVEN evaluation, even the strongest MLLM falls far below human level, and failure modes recur across different model families, so simply scaling up model size cannot eliminate these deficiencies.

In many vision-language models, the number of tokens consumed by a single image is often fixed and limited yet can carry a large amount of textual information, so converting text into a screenshot and inputting it as an image may save more tokens than direct text input.

Multimodal models typically have a token limit for a single image, requiring long documents to be processed in slices, and API call latency and cost are higher than those of local OCR engines

Encoders in multimodal architectures require pre-training in advance, and when training an open-source multimodal large model, in addition to the internal LLM, the preceding Encoder must also be specifically trained.

Alignment is a core concept in multimodal large models, and its essence is to spatially align non-textual information and convert it into a token sequence that an LLM can understand.

When processing image-based resumes with multimodal large models, text extraction and content understanding can be completed simultaneously, eliminating the two-step process of traditional OCR extraction followed by language model analysis, and reducing downstream analysis errors caused by OCR recognition mistakes.

Multimodal large models such as Gemini encode visual information into token sequences and feed them together with text tokens into a Transformer architecture for joint processing, but the Transformer context window is limited, and large amounts of low-level pixel details are lost after visual information is encoded and compressed.

处理视频通常比处理文本消耗更多的 token 和算力，视频作为提示词的成本可能比纯文本高出一到两个数量级

The paper proposes that the deficiencies of multimodal large models in spatial, embodied, physical, and temporal reasoning may all stem from the same missing capability—visual transition reasoning.