ComfyUI-INT4-Fast: Running Flux Models Smoothly on 6GB GPUs

INT4 Quantized Inference Comes to ComfyUI

For users running AI image generation models locally, VRAM has always been the biggest bottleneck. A developer recently shared on Reddit a custom node package called ComfyUI-INT4-Fast, which successfully brings INT4 (W4A4 — 4-bit weights, 4-bit activations) quantized inference into the ComfyUI ecosystem. Even a modest RTX 3060 with just 6GB of VRAM can now run large Flux-based models like Krea2 Turbo.

"Quantization" refers to compressing a model's weights and activations from 32-bit floating point (FP32) or 16-bit floating point (FP16) representations down to lower-bit integers. In W4A4, "W" stands for Weight and "A" stands for Activation — both are compressed to 4-bit integers. In theory, this reduces VRAM usage to roughly one-quarter of FP16 while dramatically increasing computational throughput. The shift from FP32 to FP16 half-precision began around 2017, when NVIDIA popularized it through mixed-precision training frameworks on Pascal/Volta architectures. INT8 quantization entered mainstream industrial deployment between 2018 and 2020 as TensorRT and ONNX Runtime matured. W4A4 INT4 represents the current compression frontier being pursued by both academia (e.g., MIT's SmoothQuant, QuIP#) and industry (e.g., vLLM, TensorRT-LLM). The core challenge lies in activation quantization (A4), which is far harder than weight quantization (W4): weights are static during inference and can be calibrated offline, whereas activations are generated dynamically at runtime and their distribution varies wildly with input — directly truncating them to 4 bits is highly prone to catastrophic precision loss. This is the fundamental reason W4A4 has long remained an academic exercise rather than an engineering reality.

The activation distribution problem is especially pronounced in Transformer architectures. Researchers have found so-called "outlier activations" in Transformers — a small number of channels can have activation magnitudes more than 100× larger than others. This makes a uniform quantization scale factor either compress normal values or fail to capture extreme ones. Methods like SmoothQuant address this by mathematically shifting the quantization difficulty from activations to weights; mixed-precision approaches simply bypass these difficult layers by processing them at higher precision. Both strategies are reflected in ComfyUI-INT4-Fast.

The project is an independent improvement built on top of BobJohnson24's ComfyUI-INT8-Fast, and relies on comfy-kitchen under the hood to invoke GPU Tensor Cores for hardware-accelerated computation of 4-bit weights and activations. Tensor Cores are co-processing units specifically designed for matrix multiply-accumulate (MMA) operations, complementing the scalar/vector computation of traditional CUDA Cores. NVIDIA introduced dedicated matrix computation units starting with the Volta architecture (2017). In the Ampere architecture (RTX 30 series, SM80), a single Tensor Core can complete an 8×16×16 INT4 matrix multiplication in one clock cycle, with native support for both INT8 and INT4 precision. For the RTX 3060 specifically, the theoretical peak INT4 Tensor Core throughput is approximately 101 TOPS, compared to roughly 12.7 TFLOPS for FP16 (including Tensor Cores) — a gap of more than 8×. Unlocking this potential requires data to be fed in specific memory layouts (e.g., NHWC format) and matrix dimensions to meet alignment requirements. This is precisely where the comfy-kitchen scheduling layer provides engineering value: it handles low-level details like data rearrangement and padding, so that higher-level model code remains hardware-agnostic, precisely routing computation to Tensor Cores and genuinely unleashing previously dormant hardware throughput. Compared to conventional FP16 or INT8 approaches, INT4 delivers significant advantages in both VRAM usage and inference speed.

Key Feature: Native Mixed-Precision Checkpoint Support

The most noteworthy technical feature of this node is its native mixed-precision checkpoint support, which directly determines whether a quantized model can maintain image quality while achieving maximum compression.

A mixed-precision checkpoint is a model file where different layers are stored at different numerical precisions. Such checkpoints are typically generated via one of two approaches: Post-Training Quantization (PTQ), which uses a small calibration dataset to profile per-layer activation distributions and automatically identifies sensitive layers to retain at higher precision; or Quantization-Aware Training (QAT), which inserts fake quantization nodes during training so the model actively adapts to low-precision constraints. For large pre-trained diffusion models like Krea2, PTQ is the typical path for generating mixed-precision checkpoints. The SafeTensors format natively supports specifying different data types (dtype) per tensor within the same file, providing convenient storage-level support.

The core insight behind this technique comes from extensive quantization research: different layers in a neural network vary significantly in their sensitivity to precision loss. In Transformer-based diffusion models, for example, input projection layers like patch projection directly influence the initial feature space representation — precision loss there gets amplified by every subsequent layer. Meanwhile, the attention and FFN modules in the middle backbone are far more robust to low precision. Quantization researchers typically use the "Hessian trace" to quantify this sensitivity — layers where the loss function has higher curvature cause larger output changes from small parameter perturbations, and therefore warrant higher-precision protection. This aligns with the mixed-precision search paradigm in QAT: concentrating the limited precision budget on the layers that most deserve protection.

Per-Layer Dynamic Routing

When running Flux-based models like Krea2, the node dynamically parses model metadata at load time and determines the compute path on a per-layer basis:

• Main backbone blocks: routed to the fastest INT4 Tensor Core path to maximize throughput;
• Sensitive layers (patch projection layers): high-sensitivity layers stored in INT8 format are automatically directed to Triton's INT8 execution path.

This design delivers two key benefits: it avoids dimension mismatch errors caused by inconsistent precision, and it reserves the limited precision budget for the layers that matter most to image quality — achieving results that a blunt "one-size-fits-all" INT4 quantization simply cannot match.

Real-World Performance: What a 6GB GPU Actually Delivers

The developer ran tests on a budget-friendly setup, making the results broadly relevant:

• GPU: RTX 3060 (6GB VRAM)
• RAM: 32GB
• Model: Pre-quantized Krea2 Turbo

Krea2 Turbo is an accelerated, distilled variant built on the Flux architecture. Flux was released by Black Forest Labs (the original Stable Diffusion team) in 2024, using a Flow Matching training paradigm in place of the traditional DDPM diffusion process, achieving significant improvements in generation quality and text adherence. The core advantage of Flow Matching over DDPM is that it learns a "straight-line" transport path from noise to image rather than DDPM's random walk — enabling convergence to high-quality results with fewer sampling steps, which naturally pairs well with low-VRAM devices. The Turbo variant further compresses inference to single-digit step counts through consistency distillation or adversarial distillation, making it one of the most suitable Flux-family models for low-VRAM hardware.

Test parameters and results:

Parameter	Value
Resolution	1024×1024
Steps	8
Sampler	Euler
Scheduler	Simple
LoRA	None
Speed	1.78 s/it
Total time	17.64 seconds

Note that the first generation requires compiling custom Triton kernels and takes somewhat longer — subsequent runs are noticeably faster. Triton is a GPU programming language and compiler open-sourced by OpenAI, designed specifically for writing high-performance deep learning kernels. The compiler's core abstraction is the concept of a "Program": developers describe computation logic in Python syntax, and Triton compiles it to PTX (NVIDIA Parallel Thread Execution) intermediate representation, which is then compiled by the driver layer to final machine code (SASS). Compared to CUDA, Triton integrates more naturally with the Python ecosystem and can automatically generate optimal tiling strategies (tile size, warp count, and other hyperparameters) for the current GPU, without requiring developers to maintain separate implementations for each GPU model.

Triton's JIT caching mechanism is worth understanding: it identifies each compiled artifact using a composite hash key that encodes the GPU architecture identifier (e.g., sm_86 for Ampere), CUDA driver version, input tensor dtype, memory layout (stride), and shape. Compiled artifacts are stored by default in ~/.triton/cache. Subsequent calls that hit the cache skip compilation entirely and load the binary directly — meaning the initial compilation cost (typically 30 seconds to a few minutes) is a one-time investment, and the cache persists across processes and sessions. Note that switching GPUs, upgrading CUDA drivers, or modifying kernel code will invalidate the cache and trigger recompilation. This is a common characteristic of JIT compilation approaches — after the initial "warm-up," stable high-speed inference follows.

For a 6GB VRAM configuration that many modern diffusion models consider insufficient, completing an 8-step 1024×1024 generation in under 18 seconds is genuinely practical — and opens up Flux-class model capabilities to users on tighter budgets.

Technical Significance and Usage Notes

The Continuing Evolution of Quantization

From FP16 to INT8, and now to INT4 (W4A4), the quantization roadmap for local inference is advancing rapidly. W4A4 compresses both weights and activations to 4 bits, dramatically reducing VRAM usage while fully leveraging the low-precision Tensor Core throughput available on Ampere and newer GPUs — where INT4 Tensor Cores offer 8× the theoretical peak throughput of FP16, a hardware capability that was almost never fully utilized in mainstream inference frameworks. The introduction of mixed-precision routing directly addresses the core concern about over-quantization degrading image quality, representing a maturation in quantization engineering from "brute-force compression" toward "fine-grained control." This trajectory closely mirrors developments in the LLM inference space with frameworks like GPTQ (Generalized PTQ) and AWQ (Activation-aware Weight Quantization): the LLM quantization community has already demonstrated that understanding per-layer activation distribution characteristics and dynamically allocating precision budgets is the key to making low-bit quantization practically viable. Diffusion model quantization tooling is rapidly adopting these lessons.

Getting Started

All project resources are open source and ready to try:

• Custom node (GitHub): github.com/viralvfx/ComfyUI-INT4-Fast
• Test model (Hugging Face): krea2_turbo_convrot_int4_fast.safetensors under comfyanonymous/int4_tests

Points to Watch

As a relatively new community project, a few caveats are worth keeping in mind: initial compilation takes a while; the solution depends on specific GPU architectures — you'll need an Ampere or newer GPU with INT4/INT8 Tensor Core support. The Turing architecture (RTX 20 series, SM75) deserves a special note: while Turing introduced Tensor Core INT8 support, its INT4 support is limited to specific sub-byte precision modes that are less fully exposed at the compute capability and driver levels compared to Ampere, with more limited PTX-level support for INT4 instructions — making it difficult to reliably reproduce the performance figures from the developer's tests. Ada Lovelace (RTX 40 series) and Hopper (H series) go further by introducing native FP8 support on top of Ampere's foundation, representing the next frontier in quantization precision. Additionally, compatibility and image quality consistency still await broader community validation, and the developer has explicitly invited feedback on issues encountered in practice.

Conclusion

ComfyUI-INT4-Fast represents another pragmatic step forward in local AI image generation toward "low cost, high efficiency." Rather than chasing extreme theoretical compression ratios, it achieves a usable balance among VRAM usage, speed, and image quality through per-layer mixed-precision engineering. This approach closely mirrors the evolution of quantization in the LLM inference space — from GPTQ to AWQ — where the lesson is clear: quantization schemes that actually work in practice never simply truncate all values to low bits. They make nuanced precision allocation decisions grounded in an understanding of model structure sensitivity. For users with entry-level GPUs who still want to run Flux-family models, open-source projects like this are continuously lowering the barrier to entry, and are well worth following.