Building a Search Engine for 1.7 Million arXiv Papers with Python

Project Overview

Recently, a developer shared their open-source project on Reddit: a mini search engine capable of indexing and retrieving over 1.7 million scientific papers. All papers come from the arXiv dataset maintained by Cornell University, covering disciplines including physics, mathematics, computer science, and more.

arXiv is an open-access preprint repository created and maintained by Cornell University since 1991. Originally focused on physics, it has since expanded to mathematics, computer science, statistics, electrical engineering, and other fields. As of 2024, arXiv has accumulated over 2.4 million papers, with more than 15,000 new submissions per month. Its open metadata and paper content provide an ideal data source for academic search engine development—arXiv offers bulk data access interfaces (OAI-PMH protocol and Kaggle datasets), allowing developers to legally obtain structured information such as paper titles, abstracts, authors, and categories to build retrieval systems.

The project's core goal is straightforward—enabling researchers to quickly locate papers by author name or keywords without manually sifting through massive document collections. The entire system is implemented in 100% Python, with FastAPI and Streamlit as the primary tech stack, maintaining clear separation between frontend and backend responsibilities.

For researchers and students who need to filter through a sea of literature daily, this kind of lightweight retrieval tool offers genuine practical value. It lowers the barrier to information access and serves as a reference implementation for learning information retrieval techniques.

Technical Architecture Analysis

The FastAPI + Streamlit Tech Stack Choice

Based on the project description, the author chose FastAPI as the backend service framework. FastAPI is a modern web framework based on Python 3.6+ type hints, released by Sebastián Ramírez in 2018. It uses Starlette as its ASGI framework and Pydantic for data validation under the hood, automatically generating API documentation compliant with the OpenAPI standard. In performance benchmarks, FastAPI's results approach those of frameworks written in Node.js and Go, far exceeding traditional Flask and Django. Its core advantage lies in native support for the async/await asynchronous programming model, enabling I/O-intensive operations (such as database queries and index retrieval) to be processed concurrently with high efficiency. When a user submits a query, the backend quickly matches against pre-built index structures and returns results to the frontend, with the asynchronous architecture ensuring stable response times even under high concurrency.

The frontend is built with Streamlit. Streamlit is an open-source application framework designed specifically for data scientists and machine learning engineers, released in 2019 and acquired by Snowflake in 2022. Its core philosophy is "script as application"—developers simply write ordinary Python scripts, and Streamlit automatically transforms them into interactive web applications without any frontend development experience required. Every user interaction (such as entering a search keyword) triggers the script to re-execute from top to bottom, with the framework using intelligent caching mechanisms (@st.cache_data) to avoid redundant computation. While this architecture isn't suitable for complex production-grade frontends, it delivers extremely high development efficiency for prototyping and internal tools. This tech stack choice allows the project to maintain the consistency of a pure Python stack while rapidly iterating toward a usable user interface.

The Core Challenge of Indexing 1.7 Million Papers

Processing over 1.7 million articles is no trivial task. Although the author didn't detail the specific indexing implementation in the original post, based on common practices in retrieval systems, the core likely involves building an Inverted Index—mapping keywords to lists of documents containing those terms, thereby avoiding full scans during queries.

The inverted index is the core data structure of virtually all modern search engines, with a principle similar to the index pages at the back of a book. In a forward index, the structure is "document → term list"; the inverted index reverses this to "term → document list." In practice, each term is associated with a posting list that records all document IDs containing that term, usually accompanied by metadata for ranking such as term frequency (TF) and position information. At the scale of 1.7 million papers, the inverted index could occupy several GB of storage. To improve query efficiency, posting lists are typically compressed using encoding schemes (such as Variable Byte Encoding or PForDelta), with skip pointers used to accelerate intersection operations for multi-keyword queries. Classic ranking algorithms like BM25 and TF-IDF rely on statistical information provided by the inverted index to calculate document relevance scores.

For author name retrieval, the system likely maintains a separate author index; for keyword retrieval, tokenization and index construction are needed for text fields such as paper titles and abstracts. At this data scale, index storage efficiency and query response time are the key factors determining user experience.

Project Value and Use Cases

A Practical Retrieval Tool for Research Scenarios

As one of the world's most important preprint platforms, arXiv sees a large volume of new papers uploaded daily. For researchers, efficiently searching through massive literature collections has always been a pain point. While this mini search engine is modest in scale, it precisely addresses this need.

By searching by author or keyword, users can quickly track a scholar's research output or aggregate related papers around a specific topic. Compared to browsing page by page on the arXiv website, a localized retrieval tool is often more flexible and efficient in specific scenarios.

An Open-Source Learning Resource for Search Engine Development

The project is open-sourced on GitHub (repository: KarimData06/mini_search_engine1), meaning anyone can view the source code, reproduce results, or even build upon it. For beginners looking to learn information retrieval, search engine principles, or full-stack Python development, this is a moderately-scoped, clearly-defined hands-on case study.

From data acquisition, text processing, and index construction to API development and frontend presentation, the project comprehensively covers the core components of a retrieval system, offering significant educational reference value.

Notable Areas for Improvement

As a personal open-source project, there's still room for improvement in certain areas. First is retrieval quality—pure keyword matching often fails to understand query intent, and introducing semantic search (such as similarity search based on vector embeddings) can significantly improve result relevance.

Semantic search represents the next-generation retrieval paradigm compared to traditional keyword matching. Its core idea is to use pre-trained language models (such as BERT, Sentence-BERT, OpenAI Embeddings) to transform text into high-dimensional vector representations, so that semantically similar texts are closer together in vector space. During retrieval, the user query is similarly encoded as a vector, and then approximate nearest neighbor (ANN) search algorithms find the most similar documents in a vector database. Common vector indexing libraries include FAISS (developed by Facebook), Annoy, and ScaNN, while vector databases include Pinecone, Milvus, and Weaviate. In academic search scenarios, semantic retrieval can understand synonyms, abbreviations, and conceptual relationships—for example, searching for "deep learning" can also return papers containing "neural network," which pure keyword matching cannot achieve. Currently, mainstream academic search engines like Semantic Scholar and Google Scholar have deeply integrated semantic retrieval capabilities. The application of embedding techniques powered by large language models is increasingly widespread in academic retrieval and represents a direction worth exploring.

Second is performance optimization. As data scales continue to grow, maintaining millisecond-level query responses requires further refinement in index structures, caching strategies, and other areas. Additionally, incorporating specialized search engine components is a common engineering choice. Elasticsearch is a distributed search and analytics engine built on Apache Lucene, supporting horizontal scaling, near-real-time indexing, and complex full-text search capabilities that can easily handle billions of documents—though it requires deploying a separate Java runtime environment and carries higher operational costs. Whoosh is a lightweight full-text search library written entirely in Python, storing all index data in the local file system without requiring additional service processes, making it ideal for small-to-medium-scale embedded search needs. For the scale of 1.7 million papers, Whoosh can generally handle the workload, but if data volume continues to grow or high-concurrency access is needed, migration to Elasticsearch or its lighter alternatives such as MeiliSearch or Typesense may be necessary.

Conclusion

This mini search engine built on the arXiv dataset demonstrates how to build a practical retrieval tool covering 1.7 million papers using a pure Python tech stack. Its architecture is clean and straightforward, its application addresses real research needs, and as an open-source project, it offers excellent learning value.

For readers looking to get started with search engine development or improve their literature retrieval efficiency, it's worth following this project's open-source repository, or using it as a starting point to explore more advanced semantic retrieval techniques. In an age of information explosion, the ability to efficiently find the knowledge you need is itself a significant form of productivity.