GPU
A Graphics Processing Unit (GPU) is a specialized parallel computing processor originally designed to accelerate computer graphics rendering by simultaneously handling multiple tasks across a large number of small computational cores. Its highly parallel architecture provides significant performance advantages in graphics display, scientific computing, deep learning training and inference, and data processing, and it is widely deployed in personal computers, servers, and data centers.
Core Facts
Continuous batching allows new requests to be inserted and completed requests to be removed at any iteration step, keeping the GPU running at full load at every step.
NVIDIA CUDA was launched in 2006 and was originally a programming framework for enabling developers to use GPUs for general-purpose computing.
The pre-training process of large language models requires thousands of high-end GPUs running in parallel for months, with costs typically ranging from tens of millions to hundreds of millions of dollars.
Timeline (last 90 days)
High-end AI chips are typically depreciated over three to five years, meaning they swallow a large chunk of profit every year.
Maia 300 falls into the ASIC (application-specific integrated circuit) category, in contrast to the general-purpose programmable architecture of Nvidia GPUs.
Major cloud vendors generally take a "two-legged" approach: GPUs for frontier exploration and general-purpose training, and self-developed ASICs for workloads that are already stable and at scale.
The advantage of GPUs lies in a mature ecosystem (CUDA), support for multiple precisions and operators, and flexible adaptation to the full range of scenarios from training to inference; ASICs, by contrast, hardwire circuits for specific workloads, offering better efficiency and cost in target scenarios but less flexibility.
Scale-to-zero can partially mitigate the seasonality problem, but cold-start latency (GPU node scheduling, image pulling, model weight loading) is typically between 30 seconds and several minutes, which is unfriendly to interactive queries.
Cosmic rays can cause single-event upsets, and commercial GPUs cannot be used directly in the space environment, requiring radiation-hardened designs or redundant verification mechanisms.
VKORE is a software acceleration layer for NPUs, designed to be hardware-agnostic, enabling data centers to mix GPUs and NPUs and still train and serve inference from the same stack.
GPUs have a large number of general-purpose compute cores and can be flexibly scheduled through programming models such as CUDA, making them naturally suited to the frequent matrix operations and gradient backpropagation in training; NPUs, by contrast, are dedicated chips customized for neural network inference, typically using systolic arrays or TPU-like dataflow architectures, highly optimized for fixed operators, but with closed programming models and limited support for dynamic computation graphs, making it traditionally difficult for NPUs to carry training workloads.
Ordered Dithering is fully parallelizable and vectorizable because each pixel depends only on its own coordinates and a fixed matrix and contains no cross-pixel dependencies, making it very suitable for per-fragment GPU computation.
VKORE is positioned as a hardware-agnostic software layer, aimed at data centers with mixed GPU and NPU deployments, and is not tied to a specific chip vendor.
40 more timeline events
All Facts (20)
Continuous batching allows new requests to be inserted and completed requests to be removed at any iteration step, keeping the GPU running at full load at every step.
90%VerifiedNVIDIA CUDA was launched in 2006 and was originally a programming framework for enabling developers to use GPUs for general-purpose computing.
90%VerifiedThe pre-training process of large language models requires thousands of high-end GPUs running in parallel for months, with costs typically ranging from tens of millions to hundreds of millions of dollars.
90%Verified训练一次GPT-4级别的模型需要数万块A100/H100 GPU协同工作数月
90%VerifiedThe NPU is a hardware accelerator designed specifically for AI inference computation; the Hexagon NPU in Qualcomm's Snapdragon 8 series, Apple's Neural Engine, and MediaTek's APU have all been integrated into mobile chips.
80%VerifiedEfficient operation of Tensor Cores requires matrix dimensions to be aligned to multiples of 16 or 8.
80%VerifiedDuring large model inference, actual GPU utilization is often less than 10%.
80%VerifiedThe massively parallel computing architecture of GPUs is naturally suited for matrix operations in deep learning, and their thousands of computing cores can accelerate the training process by tens to hundreds of times.
80%VerifiedTensorRT unleashes GPU hardware potential through operator fusion, kernel auto-tuning, and memory optimization.
80%VerifiedTensorRT's operator fusion merges multiple consecutive small operators in a computation graph into a single large operator, reducing the number of GPU kernel launches and memory reads/writes of intermediate results; a typical fusion operation can combine convolution, batch normalization, and activation function into a single GPU kernel call.
80%VerifiedTraining a frontier large model requires hundreds of millions of dollars in compute, data cleaning, and human labor.
80%VerifiedNVIDIA has long dominated the GPU computing market required for AI training and inference.
75%VerifiedThe bottleneck in large model inference is often not compute but memory bandwidth, and GPU compute utilization during the inference phase is often less than 10% of peak.
75%VerifiedFor technically capable users, fully local deployment on their own GPUs using tools such as vLLM and llama.cpp often yields a per-token cost far lower than cloud-based paid solutions in long-term, large-scale usage scenarios.
75%VerifiedCPU-GPU hybrid inference (offloading) introduces a significant drop in inference speed.
75%VerifiedNPUs are specifically optimized for low-precision (INT8/FP16) matrix multiplication and can achieve similar or higher throughput at a fraction of a GPU's power consumption when running quantized lightweight inference models.
75%VerifiedBatch inference (Batching) combines multiple requests within a short time into a single batch, which can significantly improve GPU throughput, but introduces additional queuing latency and requires trade-offs based on latency tolerance.
75%VerifiedLarge model inference is essentially a large number of matrix multiplication operations, and GPUs with thousands of parallel compute cores are well-suited for such tasks.
75%VerifiedThe GPU memory hierarchy from fastest to slowest is: registers, L1 cache/shared memory, L2 cache, global memory (VRAM).
75%VerifiedThe training and inference of generative AI require a large amount of parallel matrix operations, relying on GPU clusters or dedicated AI chips running under continuous heavy load.
75%