CUDA
CUDA (Compute Unified Device Architecture) is a parallel computing platform and programming model introduced by NVIDIA that allows developers to leverage the parallel processing capabilities of NVIDIA GPUs for general-purpose computation. It provides extension interfaces for languages such as C/C++, enabling programmers to directly schedule large numbers of parallel threads on the GPU. CUDA is widely used in high-performance computing domains including scientific computing, deep learning, image processing, and data analysis.
Core Facts
llama.cpp是由Georgi Gerganov开发的纯C/C++推理引擎，支持在纯CPU环境下运行大模型
NVIDIA CUDA was launched in 2006 and was originally a programming framework for enabling developers to use GPUs for general-purpose computing.
CUDA (Compute Unified Device Architecture) is a parallel computing platform launched by NVIDIA.
Timeline (last 90 days)
CUDA kernels involve a large amount of parallel semantics and memory hierarchy details, and this information is difficult to fully recover at the assembly level, which is the main bottleneck of AI-assisted reverse engineering today.
PTX (Parallel Thread Execution) is an intermediate instruction set architecture in the CUDA system, similar to Java's bytecode. NVCC first compiles CUDA C++ source code into PTX, which is then converted by the JIT compiler in the driver into SASS machine code for a specific GPU architecture.
NVIDIA's moat is not just the software API, but also the deep co-optimization between hardware and compiler; reverse engineering can make CUDA programs "run" on non-NVIDIA hardware, but achieving "fast and stable" operation is exponentially harder.
AI can help in areas such as decompilation assistance, code translation (for example, translating CUDA C++ to HIP or SYCL), test generation, and documentation completion, but true architecture-level reverse engineering still requires deep involvement by human experts.
Game mods usually involve memory modification, model replacement, or script injection on an existing PC version, essentially adding on top of a known architecture and API; CUDA reverse engineering, by contrast, involves closed-source compilers, proprietary instruction sets, and hardware microarchitecture, and AI is currently far from being able to complete such tasks autonomously.
VKORE aims to rebuild a CUDA-like abstraction layer on NPUs, providing a unified training and inference interface upward and adapting to different vendors' NPU instruction sets downward, so that data centers can swap accelerators like swapping storage or network cards without rewriting model code, thereby breaking single-vendor lock-in.
GPUs have a large number of general-purpose compute cores and can be flexibly scheduled through programming models such as CUDA, making them naturally suited to the frequent matrix operations and gradient backpropagation in training; NPUs, by contrast, are dedicated chips customized for neural network inference, typically using systolic arrays or TPU-like dataflow architectures, highly optimized for fixed operators, but with closed programming models and limited support for dynamic computation graphs, making it traditionally difficult for NPUs to carry training workloads.
There is a strict version correspondence between CUDA and the graphics card driver and inference engine, and version mismatch is one of the most common causes of local deployment failure.
The root cause of this bug is that the official image was built on CUDA 12.9, but when installing the low-level libraries related to audio processing it mistakenly pulled in CUDA 13 components.
The solution is to rewrite the Dockerfile based on CUDA 12.9 and reinstall the Torch-related libraries to rebuild the image.
40 more timeline events
All Facts (20)
llama.cpp是由Georgi Gerganov开发的纯C/C++推理引擎，支持在纯CPU环境下运行大模型
90%VerifiedNVIDIA CUDA was launched in 2006 and was originally a programming framework for enabling developers to use GPUs for general-purpose computing.
90%VerifiedCUDA (Compute Unified Device Architecture) is a parallel computing platform launched by NVIDIA.
90%VerifiedROCm is AMD's open-source software platform for high-performance computing and AI workloads, including the HIP programming interface, which allows CUDA code to be ported to AMD GPUs at relatively low cost.
90%VerifiedNVIDIA's CUDA ecosystem is the de facto standard in the AI inference industry.
85%VerifiedCUDA is the underlying support for virtually all mainstream deep learning frameworks.
85%VerifiedSince its introduction in 2006, CUDA has become the de facto industry standard in the AI/deep learning field, with mainstream frameworks such as PyTorch and TensorFlow using CUDA as their default backend.
80%Verifiedllama.cpp supports GPU acceleration backends including Metal, CUDA, Vulkan, and SYCL.
80%VerifiedOllama automatically detects NVIDIA GPUs in the system and selects the optimal inference backend (CUDA, Metal, or pure CPU), and achieves GPU acceleration for Apple Silicon users via the Metal API.
80%Verifiedllama.cpp is a highly optimized C++ inference engine that supports Metal acceleration for Apple Silicon, CUDA acceleration for NVIDIA, and ROCm acceleration for AMD.
80%VerifiedCUDA是NVIDIA于2006年推出的并行计算平台和编程模型
80%VerifiedTriton is an open-source project by OpenAI that enables developers to write high-performance GPU kernels using Python-like syntax, lowering the barrier to CUDA programming.
75%VerifiedCUDA is NVIDIA's proprietary platform, requiring explicit kernel function authoring; it has a steep learning curve but the greatest performance potential.
75%VerifiedAMD's ROCm and Intel's oneAPI still have significant gaps compared to CUDA in terms of ecosystem completeness, community support, and performance tuning tools.
75%VerifiedAs of 2024, the CUDA developer community has exceeded 4 million members, and software libraries optimized based on CUDA number more than 900.
75%VerifiedLarge model inference is essentially a large number of matrix multiplication operations, and GPUs with thousands of parallel compute cores are well-suited for such tasks.
75%VerifiedUnsloth has hand-written customized CUDA/Triton compute kernels for mainstream GPU architectures and rewritten the backpropagation algorithm.
75%VerifiedA CUDA stream is an asynchronous execution queue on the GPU; operations in different streams can execute concurrently and are a key mechanism for overlapping computation and data transfer.
75%VerifiedCUDA（Compute Unified Device Architecture）是NVIDIA推出的并行计算平台和编程模型
75%Verifiedllama.cpp supports GPU acceleration via CUDA for NVIDIA GPUs, ROCm for AMD GPUs, and Metal for Apple Silicon, and also supports pure CPU inference.
70%