inaturalist-clumper: A Guide to Automatic Clustering of iNaturalist Observation Data

What is inaturalist-clumper

Simon Willison (renowned developer and creator of Datasette) recently released version 0.1 of the open-source tool inaturalist-clumper. The tool's purpose is straightforward: automatically cluster nature observation records from iNaturalist into structured data that can be easily integrated into blogs or websites. After several weeks of running in production and multiple iterations, he officially released the tool to the community.

Introduction to the iNaturalist Platform

iNaturalist is currently one of the world's largest citizen science platforms, jointly operated by the California Academy of Sciences and the National Geographic Society. To date, the platform has accumulated over 200 million observation records covering more than 400,000 species.

"Citizen Science" refers to the practice of ordinary members of the public participating in scientific research data collection and analysis. This concept is not new—as early as the 19th century, the Christmas Bird Count in North America was already a classic example of citizen science. However, the widespread adoption of the internet and smartphones has fundamentally transformed the scale of citizen science. iNaturalist is one of the most successful platforms to emerge from this wave: it converts the daily observations of millions of nature enthusiasts worldwide into structured data with scientific value. According to Google Scholar, thousands of academic papers have been published based on iNaturalist data, covering research areas including invasive species monitoring, urban ecology, and phenological change tracking. Environmental protection agencies in some countries have also begun incorporating iNaturalist data as reference material for biodiversity assessments and conservation area planning.

Users photograph wildlife and plants in the field and upload them to the platform, where community members and AI algorithms collaboratively complete species identification. The platform's built-in computer vision model, trained on large-scale image datasets, can instantly suggest species when users upload photos, which are then confirmed or corrected by professionals and experienced enthusiasts in the community. This human-AI collaborative identification mechanism is one of iNaturalist's core competitive advantages: the AI model quickly narrows down candidates while human experts make the final judgment, with both cross-validating each other to ensure both efficiency and accuracy. When an observation record gains sufficient community consensus, it is marked as "Research Grade," and these high-quality data are synchronized to open data platforms like the Global Biodiversity Information Facility (GBIF), providing crucial support for global ecological research and conservation decisions.

iNaturalist provides a fully-featured RESTful API that supports querying observation records across multiple dimensions including user, species, geographic range, and time period, returning JSON data with rich fields including species taxonomy, GPS coordinates, photo URLs, and identification status. A RESTful API is an interface design style based on the HTTP protocol that uses standard request methods like GET and POST to operate on resources, returning structured data formats (typically JSON). iNaturalist's API design follows good REST practices, supporting pagination, filtering, sorting, and other common operations, and access to public data requires no authentication, greatly lowering the barrier for third-party developers. Many developers and nature enthusiasts use this API to build personalized display tools or data analysis applications. Simon Willison's inaturalist-clumper is a noteworthy example of this kind of practice.

Core Functionality and Clustering Mechanism

The core capability of inaturalist-clumper lies in its clumping of iNaturalist observation records. Specifically, it automatically groups multiple scattered observation records based on temporal and geographic proximity, forming meaningful "observation sets."

This spatiotemporal proximity-based clustering approach is very common in geographic information processing. The basic logic is: if two observation records are close in time (e.g., within the same day) and their geographic coordinates are within a certain radius, it's reasonable to infer they belong to the same field trip. Common implementation approaches include simple threshold-based grouping and algorithms like DBSCAN (Density-Based Spatial Clustering).

DBSCAN (Density-Based Spatial Clustering of Applications with Noise) is a classic clustering algorithm proposed by Martin Ester et al. in 1996. Its core idea is to find high-density connected regions separated by low-density areas in data space. The algorithm requires only two parameters—neighborhood radius (ε) and minimum points (MinPts)—without needing to pre-specify the number of clusters, and it naturally handles noise points (outliers that don't belong to any cluster). These properties make DBSCAN particularly suitable for geographic data processing: GPS trajectory analysis, point-of-interest aggregation, and travel pattern recognition are all typical applications. However, DBSCAN performs poorly on datasets with significant density variations, and parameter selection significantly affects results, requiring some tuning experience.

inaturalist-clumper uses a relatively intuitive threshold approach—setting time windows and distance thresholds to group records meeting the criteria into the same "clump." Compared to more complex algorithms like DBSCAN, this approach has the advantage of transparent logic and easy debugging, allowing users to intuitively understand why certain records are grouped together. For the iNaturalist scenario—where users typically record observations intensively during a single outing with obvious spatiotemporal gaps between different outings—a simple threshold approach often achieves sufficiently good results without introducing heavier algorithmic dependencies. This way, you don't need to manually organize each record; instead, you directly receive data packages already grouped by trip or area.

The tool ultimately outputs a JSON file stored in a GitHub repository. This design continues Simon Willison's consistent technical preference—managing data with plain file formats plus Git version control, without introducing additional database dependencies, keeping deployment and maintenance costs very low. This pattern is sometimes called "Git-as-a-database," and its core advantage lies in: every data change automatically gets a complete version history, enabling precise tracking of state at any point in time; JSON files are plain text, naturally suited for diff comparison and code review; meanwhile, GitHub repositories come with free CDN distribution and API access capabilities, allowing other applications to fetch the latest data directly via raw file URLs without needing to set up any backend service.

The "Git-as-a-database" pattern has several successful implementations in the industry. For example, Netlify CMS (now renamed Decap CMS) stores content directly as Markdown and JSON files in a Git repository; many open-source projects' configuration management (such as Kubernetes GitOps practices) also uses Git as a "Single Source of Truth." The limitations of this pattern are equally worth noting: it's unsuitable for high-frequency write scenarios (Git's merge mechanism cannot handle concurrent write conflicts), doesn't support complex queries (lacking SQL-like indexing and aggregation capabilities), and when data volume grows beyond a certain scale, repository size and clone speed become bottlenecks. But for a scenario like inaturalist-clumper—with low update frequency (at most once per day) and manageable data volume (personal observation records typically in the thousands)—Git-as-a-database is a perfectly appropriate choice.

For developers who need to handle iNaturalist data clustering, this lightweight approach saves the time of writing clustering logic from scratch.

Simon Willison's Open-Source Tool Design Philosophy

Although inaturalist-clumper is small in scope, it reflects several core principles of Simon Willison's approach to developer tools. It's worth noting that Simon Willison is not an ordinary independent developer—he is one of the co-creators of the Django web framework, having participated in Django's early development as far back as 2005, an experience that profoundly shaped his understanding of software design. After leaving the Django core team, he shifted his focus to the data tools space, creating the Datasette project and building a vast open-source tool ecosystem around it. He is also one of the most active practitioners and documenters in the AI-assisted programming space, and his blog (simonwillison.net) is an important source for tracking LLM toolchain developments. Understanding this background helps us better appreciate the design decisions behind inaturalist-clumper.

Production Validation First

In his release notes, he mentioned that this tool had been running in his own production environment for several weeks and went through repeated adjustments before the 0.1 version was released. This "dogfooding" approach (using it yourself first) ensures the tool has sufficient stability when officially released.

The term "dogfooding" originated from Microsoft's internal saying in the 1980s—"eating your own dog food," meaning using your own products internally first. This practice has become an important quality assurance method in the software industry: when developers themselves are daily users of the tool, edge cases that documentation would never cover and rough spots in the user experience naturally surface during real use. Google's Chrome browser, Stripe's payment API, and other well-known products all incorporate dogfooding as a mandatory step before release. Simon Willison applies this principle to personal projects, ensuring even a small tool undergoes thorough real-world validation before public release.

Combining Small Tools to Build Automated Workflows

inaturalist-clumper is responsible for only one step in the blog publishing workflow—data clustering. Simon Willison prefers to chain together multiple small tools, each with its own specific responsibility, to build flexible and easy-to-maintain automation pipelines. This is directly aligned with the Unix philosophy of "each program does one thing well."

The Unix philosophy was originally proposed by Ken Thompson and Dennis Ritchie in the 1970s and later systematically summarized by Doug McIlroy into several core principles: each program does one thing well; programs collaborate through text streams; composition is preferred over building monolithic systems. This philosophy profoundly influenced subsequent software engineering practices, from command-line pipes to microservice architectures. Simon Willison's entire tool ecosystem is a modern interpretation of this philosophy—he built dozens of plugins and companion command-line tools around Datasette (a tool that instantly transforms SQLite databases into interactive APIs and web interfaces), each solving a specific problem and seamlessly connecting through standardized data formats (SQLite databases, JSON files).

Datasette itself is the masterwork of this philosophy. Its core concept is extremely concise: give it a SQLite database file, and it immediately provides a web interface with search, filtering, and faceted navigation functionality, while automatically generating corresponding JSON APIs. This means any tool that can output a SQLite database can immediately gain a fully functional data browsing and querying interface. Simon Willison has built numerous "data importers" around this core—from Twitter archives, GitHub activity, Apple Health data to iNaturalist observation records, various data sources have corresponding tools to convert them into SQLite format. inaturalist-clumper is yet another link in this tool chain; its processed JSON data can be further imported into Datasette for interactive exploration or directly consumed by static site generators.

Open-Sourcing Even Personal Projects

Even infrastructure tools developed for personal needs are open-sourced. The project is hosted on GitHub, with code and output examples fully public, making it easy for developers with similar needs to directly reuse or reference. This "open by default" approach is not uncommon in the independent developer community, but what sets Simon Willison apart is his emphasis on documentation and discoverability—every project has a clear README, release notes, and usage examples, lowering the barrier for other developers to get started. He has articulated his view multiple times on his blog: open source is not just about putting code on GitHub; more importantly, it's about enabling others to understand, use, and contribute.

Use Cases for inaturalist-clumper

If you're both an iNaturalist user and a developer with some technical background who wants to showcase nature observation records on your blog or website, inaturalist-clumper provides an out-of-the-box solution. The following scenarios are particularly well-suited:

• Automated nature blog publishing: Periodically auto-generate iNaturalist observation summaries, eliminating the repetitive manual organization work
• Data visualization projects: Pre-process observation data into structured JSON for direct integration with frontend charting libraries (like D3.js, ECharts) or map components (like Leaflet, Mapbox), quickly building species distribution heatmaps or timeline visualizations
• Static website content updates: Combined with CI/CD tools like GitHub Actions, achieve fully automated building and deployment of nature observation content

GitHub Actions is GitHub's built-in automation engine that allows developers to define trigger conditions (such as scheduled execution or code pushes) and execution steps through YAML configuration files. In a typical inaturalist-clumper workflow, you can set up a cron job to automatically run the clustering script daily, commit the generated JSON file to the repository, then trigger a static site generator (like Hugo or Eleventy) to rebuild pages and deploy to hosting platforms like GitHub Pages or Netlify. The entire process requires no manual intervention—from data acquisition to page publication, everything is fully automated.

This combination of "data source + static generator + automated deployment" is a typical practice of the Jamstack architecture in recent years. Jamstack (an abbreviation of JavaScript, APIs, and Markup) is a modern web development architecture whose core concept is decoupling the frontend presentation layer from the backend data layer: pages are generated at build time rather than request time, with dynamic data pulled via APIs during the build phase and pre-rendered as static HTML. The advantages of this architecture include: extremely fast page load speeds (pure static files distributed via CDN), minimal operational costs (no server management needed), and inherent security (no dynamic server means a drastically reduced attack surface). Hugo, Eleventy, Next.js, Astro, and others are mainstream static site generators in the Jamstack ecosystem. The JSON files output by inaturalist-clumper are naturally suited to this architecture—serving as the "API data source" role in Jamstack, providing structured content to static generators during the build phase.

Summary

inaturalist-clumper 0.1 is a quintessential Simon Willison-style project—small, focused, and ready to use. It seamlessly connects personal interests (nature observation) with technical capabilities (automated publishing) through a concise tool chain, while also contributing a reusable iNaturalist data clustering component to the open-source community.

Whether you're interested in automated processing of nature observation data or automated blog content publishing workflows, this tool is worth adding to your toolbox.