DeepSeek Open-Sources Inference Framework, Shifting the AI Compute Landscape

AI infrastructure is undergoing a dense round of foundational changes. From open-sourcing inference frameworks to breaking down hardware ecosystem barriers, from filling in local office capabilities to launching ultra-long cache mechanisms, nearly every one of these is rewriting how developers and enterprises use large models. Below is a review and analysis of several recent key changes.

DeepSeek Open-Sources Inference Framework, Bringing Low-Level Acceleration Technology Into the Open

According to a related video on Bilibili, DeepSeek^Ad announced that it is fully open-sourcing the inference framework it uses internally to run large-scale model services, and is simultaneously opening its DeepSeek Recipe codec components. This means developers and enterprises can directly access the same low-level acceleration technology used by DeepSeek, substantially reducing the compute, cost, and energy consumption of large models.

It is worth noting that this framework is primarily aimed at enterprise cluster environments and has specific requirements for task configuration and quantization environments on compute platforms such as NVIDIA and Ascend. Open source does not mean out-of-the-box usability; real deployment still requires a certain amount of engineering adaptation. But from an industry perspective, opening up core inference-side components helps narrow the efficiency gap between small and medium teams and large companies.

Inference framework refers to software systems specifically designed to efficiently run already-trained large models. Its core tasks include computation graph optimization, operator fusion, memory management, batch scheduling, and quantized inference. Unlike training frameworks (such as PyTorch), inference frameworks focus more on low latency, high throughput, and low-cost deployment. Common inference frameworks in the industry include vLLM, TensorRT-LLM, SGLang, and others, which significantly improve serving efficiency through techniques such as PagedAttention, continuous batching, and speculative decoding. The framework DeepSeek is open-sourcing this time is precisely the inference engine that internally supports its large-scale online services. After opening it up, developers can learn from its engineering optimization ideas, but they should note that different hardware platforms (such as NVIDIA GPUs and Ascend NPUs) differ in operator libraries and quantization schemes, so adaptation and verification are needed during migration.

Doubao Completes the Office Suite, Enabling Direct Local Editing in Word

ByteDance's Doubao has officially added support for Word documents in local Office scenarios, completing its office suite. Users can right-click a local Word file to invoke AI for precise rewriting and formatting polish based on context, and save changes in place after editing, with no upload, download, or format conversion required.

The substantive significance of this change is that it embeds AI editing capabilities into users' existing file workflows, rather than making users shuttle back and forth between web pages and local files. However, the video also notes that using it requires installing the latest desktop client, and modifications directly overwrite the original document, so it is advisable to keep a copy of important documents first. Balancing convenience and data security remains an ongoing challenge for local AI office work.

OpenAI Launches 12-Hour Prompt Cache, Halving the Cost of Long Tasks

OpenAI announced that its API has officially launched an optional 12-hour ultra-long Prompt cache, while drawing cache for logged-in Codex and Work users is retained permanently. For long-running agents, even calls made several hours apart can reliably hit the cache, directly halving input costs and delivering near-zero latency.

The 12-hour long cache must be actively enabled when calling the API, and ordinary anonymous conversations do not enjoy permanent retention. This mechanism is especially critical for developers building continuously running agents—higher cache hit rates translate directly into lower costs and latency, and also significantly enhance the engineering feasibility of long-cycle tasks.

Prompt caching is a mechanism for reducing the cost and latency of large model API calls. Its principle is that when the prefix content of multiple requests (such as system prompts, long documents, or conversation history) is exactly the same, the server reuses the already-computed key-value cache (KV Cache), avoiding repeated forward computation. This not only greatly reduces billing for input tokens (usually the cached hit portion is billed at a discounted price), but also significantly lowers first-token latency. OpenAI previously offered short-term caching, and extending the validity period to 12 hours this time means that long-running agents can still hit the cache when called again after several hours. It is usually enabled by setting specific parameters in the request, and caching only applies to identical prefixes, so designing prompt structure reasonably (placing stable content first) is key to improving hit rates.

Google and Meta Jointly Open-Source Torch TPU, Breaking CUDA's Exclusive Lock-In

Google and Meta have made an unprecedented joint effort to open-source Torch TPU, directly building a native TPU hardware backend for PyTorch, with Hugging Face also providing full support simultaneously. Developers only need to change a small amount of code in their existing PyTorch code to run models seamlessly on Google TPU.

From an ecosystem perspective, this move breaks AI compute's exclusive lock-in to NVIDIA CUDA and provides developers with more hardware choices. The initial release focuses on adapting to the TPU V5 series architecture, and complex custom operators still need to be migrated gradually. In the short term, it may not be able to shake CUDA's dominant position, but having one more viable path itself reduces supply chain risk for the entire industry.

CUDA is the parallel computing platform and programming model introduced by NVIDIA. After more than a decade of ecosystem accumulation, it has become the de facto standard for deep learning training and inference. A large number of deep learning frameworks and operator libraries prioritize CUDA support, creating strong ecosystem lock-in. TPU (Tensor Processing Unit) is Google's self-developed dedicated AI accelerator chip, previously accessed mainly through JAX and TensorFlow and not friendly enough for PyTorch users. The Torch TPU project aims to provide a native TPU backend for PyTorch so that developers can run models on TPU without rewriting code. This is similar to PyTorch's support model for other hardware backends (such as ROCm and XPU). Breaking exclusive lock-in helps reduce hardware procurement risk and promote price competition in compute, but operator coverage and performance tuning for the new backend still need time to mature, and CUDA's ecosystem advantage will remain solid in the short term.

Perplexity Computer Supports Fully Local Offline Operation on Mac

The CEO of Perplexity announced that its agent product Perplexity Computer now supports fully local offline operation on Mac. Even if the network cable is completely unplugged and WiFi is not used, it can directly schedule files and native applications locally, and company secrets and private data are never uploaded to the cloud.

This feature requires the latest Mac client and Apple M-series chips; older Intel processors cannot run it. The value of local offline agents lies in keeping the processing of sensitive data on the device, which is attractive to privacy-sensitive industries such as finance and legal services, but the hardware threshold also means coverage remains limited.

Local offline operation of an agent means that model inference, file indexing, and application scheduling are all completed on the user's device without relying on cloud APIs. This usually requires an on-device model (with fewer parameters or quantized compression) paired with a local inference engine (such as llama.cpp, MLX, etc.). Apple M-series chips use a unified memory architecture, where the CPU and GPU share high-bandwidth memory, making them suitable for running medium-scale quantized models and therefore a popular platform for local AI. The advantages of offline mode are that data does not leave the device, privacy risk is low, and there is no network dependency; the limitation is that model capability is usually weaker than cloud-based large models and is constrained by device memory and compute. For sensitive scenarios such as finance and legal services, local agents can serve as a compliance-friendly supplementary solution, but the hardware threshold (such as supporting only M-series chips) also limits their pace of adoption.

SpaceX AI Compute Rental Revenue Surpasses Starlink and Rocket Launch

SpaceX's latest earnings call disclosed that AI compute rental generated $7.6 billion in revenue in a single quarter, accounting for more than half and surpassing Starlink and rocket launch for the first time to become its largest business. The supercomputing center built by Musk not only serves internal needs but also directly provides compute to Code, Swappy, Google, and others.

This data indirectly confirms the explosive growth in compute demand. But it should also be noted that its single-quarter capital investment reached $16.4 billion, clearly constrained by power and chip supply cycles. The scale effect of the compute business coexists with pressure from capital expenditure, and whether it can continue to work sustainably still depends on how quickly supply-side bottlenecks are alleviated.

Conclusion

These changes all point to the same trend: AI competition is sinking from the model capability layer to the infrastructure layer. Open-source inference frameworks, diversified hardware backends, optimized caching mechanisms, and local offline operation are all paving the way for more efficient and more controllable AI applications. For developers and enterprises, paying attention to these foundational changes is often more valuable in the long run than chasing updates to individual model versions.

One-line Summary

AI infrastructure is undergoing dense changes: open-source inference frameworks, TPU barrier-breaking, long caching, and local offline operation are in focus.

Paragraph Summary

Recently, several key changes have emerged at the AI infrastructure layer: DeepSeek open-sourced its internal inference framework and Recipe codec components, reducing the compute cost of enterprise large-model deployment but requiring engineering adaptation; Doubao completed its Office suite, supporting AI rewriting of local Word documents, with convenience and data security needing to be balanced; the OpenAI API launched a 12-hour ultra-long Prompt cache, halving input costs for long tasks and delivering near-zero latency; Google and Meta jointly open-sourced Torch TPU, providing a native TPU backend for PyTorch and breaking CUDA's exclusive lock-in; Perplexity Computer supports local offline operation on Mac, meeting privacy-sensitive needs but with a high hardware threshold; SpaceX AI compute rental revenue reached $7.6 billion in a single quarter, surpassing Starlink and rocket launch for the first time, but capital expenditure pressure is enormous. The overall trend is that AI competition is sinking from the model capability layer to the infrastructure layer.