Tianshu: Open-Source AI Document Parsing Platform — A One-Stop Solution for PDF to Markdown Conversion

Project Overview: Why Data Preprocessing Is the Biggest Bottleneck in AI Deployment

In the process of deploying large language model applications, data preprocessing has always been the most underestimated yet most critical step. According to multiple industry surveys, 60%-80% of time in AI projects is actually spent on data collection, cleaning, and preprocessing. The principle that "data quality determines the model's upper bound" is widely acknowledged in the industry. Most enterprise knowledge assets are stored in unstructured formats like PDF, Word, and PPT — document formats originally designed for human reading, not optimized for machine understanding. How to efficiently convert massive volumes of PDF and Office documents into structured data consumable by large models is an unavoidable challenge for every AI engineering team.

PDF format is particularly complex — it's essentially a Page Description Language that stores absolute coordinate positions of characters rather than logical structure, making semantic information extraction an extremely challenging engineering task. Tianshu (mineru-tianshu) was born to address this exact pain point — an enterprise-grade AI one-stop data preprocessing platform.

The project has already earned 624 Stars on GitHub, is developed in Python, and provides a complete workflow from document parsing to multimodal information extraction. It's one of the few full-stack document processing solutions available in the open-source community.

Core Capabilities: What Can Tianshu Do

PDF/Office to Markdown: From Unstructured to Structured

Tianshu's core function is converting PDF and Office documents (Word, Excel, PPT, etc.) into Markdown format. While this capability seems simple on the surface, it involves numerous technical challenges:

• Complex Layout Recognition: Multi-column layouts, nested tables, and mixed text-image arrangements in PDF documents require precise layout analysis capabilities. Modern Layout Analysis typically employs deep learning-based object detection models (such as the YOLO series, Faster R-CNN, etc.) to identify different region types within a page — titles, body text, tables, images, headers and footers, etc. Beyond that, Reading Order Detection is needed to determine the logical reading sequence of each region, which is particularly difficult in multi-column layouts and complex mixed content scenarios.
• Format-Faithful Conversion: Preserving the hierarchical structure, heading relationships, and semantic information of the original document as much as possible during conversion
• Multimodal Information Extraction: Processing not just text but also recognizing and extracting images, tables, formulas, and other multimodal content within documents. Table recognition requires additional Table Structure Recognition technology to restore visual tables into row-column structured data.

The choice of Markdown as the output format is quite wise — it's one of the most LLM-friendly input formats while retaining sufficient structural information for downstream applications like RAG (Retrieval-Augmented Generation). There are multiple technical reasons why Markdown has become the preferred intermediate format in LLM data pipelines: mainstream large models (GPT, Claude, Llama, etc.) have substantial Markdown-formatted text in their pre-training corpora (from GitHub, technical blogs, etc.), giving models a natural understanding of Markdown syntax; Markdown's heading hierarchy (#, ##, ###) naturally corresponds to a document's semantic structure, facilitating section-based document chunking for RAG systems; compared to HTML or LaTeX, Markdown has less markup noise and doesn't waste precious Context Window space; additionally, Markdown format is easy to manually verify and edit, reducing the cost of data quality auditing.

MCP Protocol Integration: Let AI Assistants Directly Call Document Parsing

Tianshu supports the MCP (Model Context Protocol), a standard protocol receiving significant attention in the current AI tool ecosystem. MCP was introduced by Anthropic in late 2024, aiming to solve the standardized connection problem between AI models and external tools/data sources. Before MCP, every AI application that needed to integrate external tools required custom adapter code, resulting in extensive redundant work and fragmented integration solutions. MCP adopts a client-server architecture and defines unified specifications for tool description, parameter passing, and result returning — similar to how USB protocol standardized peripheral connections, MCP is standardizing AI tool connections.

Through MCP protocol integration, Tianshu can be directly invoked as a tool by AI assistants, enabling:

• AI assistants automatically calling document parsing capabilities
• Seamlessly completing document processing within conversation flows
• Interfacing with tool-calling mechanisms of mainstream large models like Claude and GPT

The MCP ecosystem is currently expanding rapidly, with thousands of MCP server implementations covering database queries, file operations, API calls, and various other scenarios. This means developers can embed document parsing capabilities into existing AI application architectures without writing complex glue code, significantly reducing integration costs. Tianshu's native MCP support allows it to be directly discovered and invoked by all MCP-compatible AI clients (such as Claude Desktop, Cursor, and various AI Agent frameworks).

Vue3 + FastAPI Full-Stack Architecture: Balancing Performance and Development Efficiency

In terms of technology choices, Tianshu adopts today's mainstream full-stack approach:

• Frontend: Vue3 framework, providing visual interfaces for document upload, parsing preview, and management
• Backend: FastAPI framework, delivering high-performance asynchronous API services. FastAPI is a modern Python-based web framework built on Starlette (async web framework) and Pydantic (data validation library), and has become one of the fastest-growing web frameworks in the Python ecosystem
• Python Ecosystem: Fully leveraging Python's rich AI/ML ecosystem to integrate various document parsing and model inference capabilities

FastAPI has several particularly important features for AI applications: native async support (async/await) that efficiently handles I/O-intensive document parsing tasks without blocking other requests while waiting for OCR or model inference; automatic OpenAPI documentation generation for frontend-backend collaboration and API testing; type annotation-driven request validation that reduces runtime errors; performance close to Node.js and Go web frameworks, far exceeding traditional Flask and Django. In enterprise-grade document processing scenarios, FastAPI's async characteristics enable it to handle multiple document upload and parsing tasks simultaneously, and combined with task queues (like Celery), it can achieve an elastic distributed processing architecture.

The advantage of this architecture lies in its high development efficiency, strong community support, and native async processing support, capable of handling enterprise-level concurrent document processing demands.

Typical Use Cases: Who Needs Tianshu

Enterprise Knowledge Base and RAG System Development

For enterprises with large volumes of historical documents, Tianshu can batch-convert PDF reports, contracts, technical manuals, and more into structured data, providing high-quality data sources for enterprise knowledge bases and RAG systems. This is currently one of Tianshu's most core use cases.

RAG (Retrieval-Augmented Generation) is the most mainstream architectural pattern for enterprise-level LLM applications today. Its core approach is: preprocess enterprise private knowledge documents and store them in a vector database; when a user asks a question, first retrieve relevant document fragments, then provide them as context for the large model to generate answers. RAG system effectiveness is highly dependent on data preprocessing quality — if the document parsing stage loses critical information, destroys semantic structure, or introduces noise, subsequent vector retrieval and generation stages will suffer cascading effects. The industry calls this the "Garbage In, Garbage Out" problem. Specifically, document chunking strategies (by paragraph, by semantics, by fixed length), metadata preservation (heading hierarchy, source page numbers), and how tables and images are handled all directly impact RAG system retrieval accuracy and answer quality.

AI Training Data Preparation

In large model fine-tuning and training scenarios, high-quality text data is crucial. Tianshu's document parsing capabilities can help teams extract clean, structured training corpora from various documents, saving enormous amounts of manual cleaning time. Whether performing domain-adaptive Continual Pre-training or task-specific Instruction Tuning, clean and well-formatted text data is a fundamental prerequisite.

Intelligent Document Processing Automation Pipelines

Combined with the MCP protocol, Tianshu can serve as a core component of intelligent document processing pipelines, enabling end-to-end automation from document upload, parsing, and information extraction to structured output. As AI Agent architectures become increasingly mature, Tianshu can serve as a key node in the Agent tool chain, allowing AI to autonomously decide when to invoke document parsing capabilities for higher-level process automation.

Comparison with Similar Open-Source Projects: What Differentiates Tianshu

Currently in the open-source community, there are several well-known projects in the document parsing space, such as MinerU, Marker, Docling, and others. The open-source document parsing field is in a period of rapid development: MinerU (developed by the OpenDataLab team) focuses on high-precision PDF parsing with a pipeline architecture integrating layout detection, OCR, formula recognition, and multiple other models; Marker emphasizes fast PDF-to-Markdown conversion with a focus on speed and ease of use; Docling (developed by IBM Research) provides a unified document understanding framework supporting multiple input formats. Additionally, tools like Unstructured, PyMuPDF4LLM, and LlamaParse each have their own strengths.

Tianshu's differentiated positioning lies in:

Comparison Dimension	Tianshu	Other Parsing Tools (e.g., Marker, Docling)
Product Form	Complete frontend-backend platform	Mostly CLI tools or SDKs
MCP Protocol	Native support	Typically not supported
Enterprise Features	Concurrent processing, task management	Focused on single-file parsing
Ease of Adoption	Visual interface, ready out-of-the-box	Requires development background

In short, Tianshu takes a platform-oriented approach rather than being purely a parsing tool. Most existing open-source tools are positioned as SDKs or command-line tools suitable for developer integration in code, but they lack visual management interfaces for business users and enterprise-grade task scheduling capabilities. Tianshu's platform positioning fills the gap between "developer tools" and "enterprise-grade products," giving it a stronger advantage in enterprise deployment scenarios.

Conclusion: Is Tianshu Worth Using

The Tianshu project targets the critical "data preprocessing" step in AI application deployment, providing a complete solution from document parsing to AI integration. The growth to 624 Stars also reflects strong community demand for this type of tool.

As large model applications move from demos to production, the importance of data preprocessing platforms will only increase. If Tianshu can continue to optimize parsing accuracy, format coverage, and processing performance, it has the potential to become a significant component in enterprise AI data pipelines.

If you're building a RAG system or enterprise AI knowledge base, Tianshu is an open-source solution worth prioritizing in your evaluation. The project can be found by searching for mineru-tianshu on GitHub.

Key Takeaways

• Tianshu is an enterprise-grade AI data preprocessing platform whose core capability is efficiently converting PDF/Office documents into Markdown format
• It supports MCP protocol integration and can be directly invoked as a tool by AI assistants, reducing integration costs with LLM applications
• It adopts a Vue3 + FastAPI full-stack architecture, balancing development efficiency with production-environment concurrent processing needs
• Applicable to enterprise knowledge base development, AI training data preparation, and intelligent document processing workflows
• Its differentiated positioning lies in its platform-oriented approach and native AI integration capabilities, rather than being merely a document parsing tool