CrawlOS + Supabase: A 14-County Real Estate Data Pipeline with Automated Lead Scoring in Practice

Title

CrawlOS + Supabase: A 14-County Real Estate Data Pipeline with Automated Lead Scoring in Practice

Meta Description

Breaking down a real estate data pipeline built on CrawlOS and Supabase: covering 14 Houston-area counties, using pg_cron for nightly HCAD scrapes, a 17-table schema design, and lead scoring logic that yields roughly 125 qualified leads per run.

Article Content

Scaling real estate wholesaling isn't about thicker spreadsheets—it's about having data infrastructure that actually runs. One developer has publicly shared a production-grade implementation: integrating CrawlOS into a self-hosted Supabase, covering 14 counties in the Houston metro area, scheduling nightly runs with pg_cron, scraping HCAD (Harris County Appraisal District) data, and producing roughly 125 qualified leads per run.

The full breakdown of this setup includes the 17-table schema design, the actor deployment pattern, lead scoring logic, and the pitfalls encountered moving from prototype to production. If you're building a vertical real estate tech stack, or trying to scale a data pipeline from "single-county scraping" to multi-county, this architecture is worth examining layer by layer.

Why single-county pipelines collapse at multi-county scale

Most real estate data projects start out as "one county, one table, one script." Harris County, as the most populous county in Texas, can already generate substantial leads on its own—but the problem hits the moment you try to scale: the 14 counties' appraisal districts differ in data structure, field naming, and update cadence, and the hardcoded field mappings and deduplication logic that worked before instantly break down.

The author's core judgment is to reframe the problem from "writing scrapers" to "designing data infrastructure." CrawlOS handles the scraping and parsing abstraction layer, Supabase handles relational modeling, permissions, and realtime capabilities, and pg_cron handles orchestration. Each of the three layers has its own job, so adding or removing counties no longer means rewriting business logic.

What is real estate wholesaling?

Real estate wholesaling is a low-capital arbitrage model in real estate investing: the operator (wholesaler) doesn't actually buy the property, but instead signs a purchase contract with a motivated seller, acquires the "contractual interest" in the property, and then assigns that contract at a markup to an end buyer (usually a fix-and-flipper or a rental investor), pocketing the spread (the assignment fee). The core competency isn't construction or holding—it's consistently and cheaply finding sellers willing to sell low. This explains why data pipelines become the bottleneck in this business: whoever can cover more counties at a lower marginal cost and surface higher-quality potential sellers has the scale advantage. Traditional wholesaling teams rely on driving for dollars, direct-mail postcards, and manually dialing public records, which has a high per-lead cost and is hard to replicate across regions. Appraisal district public data, by contrast, is a free and structured alternative source, making automated scraping feasible.

HCAD and the Texas appraisal district system

The Harris County Appraisal District (HCAD) is one of Texas's 254 independently established county appraisal agencies, responsible for valuing real and personal property in its county for local property tax collection. Texas operates a "county home rule" style appraisal system, which means each county has its own website, data export format, field naming, and update cadence, with no unified statewide API between them. As the appraisal district for the most populous county, HCAD has the largest data volume and relatively high public accessibility, often used as the first data source for validating this kind of pipeline. This is exactly the root of the difficulty in scaling across counties: field mappings, how the owner of record is described, exemption type codes, and parcel numbering rules all differ by county, and any county's redesign or anti-scraping policy change can break hardcoded parsing logic. Understanding this system is the premise for understanding why this article emphasizes an "abstraction layer" rather than "a stronger scraper."

The design philosophy behind the 17-table schema

The entire system is built around 17 tables, covering entities like property master records, address standardization, owner information, transaction history, lead status, and scoring snapshots. The reason for splitting things this finely is that appraisal district data is inherently highly redundant: the same property may appear across different scrape batches, and the same owner may be linked to multiple addresses.

Relational modeling liberates "leads" from one big wide table—a lead is defined as the result of a cross-table query, not a single row. That way, when scoring rules change, you don't need to backfill historical data—you just recompute the view. For teams that need to track wholesaling opportunities over the long term, this design is far more maintainable than a single-table approach.

The role of Supabase and pg_cron

Supabase is an open-source backend platform built on PostgreSQL, offering identity authentication, row-level security (RLS), auto-generated REST/realtime APIs, and object storage on top of native Postgres. For this setup, the key is that it preserves full Postgres capabilities—you can use foreign keys, views, materialized views, and triggers to model the many-to-many relationships between properties and owners, without having to compromise with NoSQL's weak constraints or a pure ORM's abstraction leaks. pg_cron is Postgres's scheduled task extension, registering jobs directly inside the database using cron syntax, so scheduling logic lives alongside the data itself, with no need to introduce external orchestration components like Airflow or Celery. Nightly window triggering, per-county staggering, and failure retries can all be described at the database layer, compressing the operational surface into a single tech stack—this is the technical foundation for the author's emphasis on "three layers, each with its own job."

Actor deployment and pg_cron nightly scheduling

On the scraping side, CrawlOS's actor pattern is used for deployment: each county and each type of data source corresponds to an independent actor, and scrape results are uniformly written to Supabase. Scheduling is handled by pg_cron, triggering scraping tasks in batches within nighttime windows to avoid peak traffic hours on appraisal district sites.

The batching strategy is what's worth paying attention to here. If all 14 counties scrape concurrently at the same moment, you're likely to get rate-limited and also create write-side spikes. The author's approach is to stagger by county and preserve retry and checkpoint-resume capability within each actor—this is the most critical change when moving from prototype to production.

How lead scoring logic works

Lead scoring inputs come from multiple dimensions: property status, owner type (whether out-of-state or institutionally held), years held, address standardization confidence, and more. Scoring results are persisted as snapshots rather than overwriting old values, making it easy to trace back "why this lead was judged high-value at the time."

The figure of roughly 125 qualified leads per run itself illustrates the filtering intensity—the raw scrape volume is far larger, and the scoring layer bears the primary noise-reduction responsibility. For a wholesaling business where lead quality matters more than quantity, this layer's value is often underestimated.

A few pitfalls from prototype to production

The pitfalls of self-hosted infrastructure cluster in three places. First, anti-scraping measures and page structure changes on appraisal district sites like HCAD—actors need a fallback path when parsing fails. Second, database connection and concurrent write stability under nightly batch jobs. Third, address standardization—across 14 counties, variations in how the same property's address is written directly pollute deduplication results.

The general lesson from this setup is: split "scrape success rate" and "lead accuracy rate" into two independent monitoring metrics. The former reflects pipeline health, the latter reflects data quality; mixing them together masks where the real problem lies.

For teams evaluating whether to build a similar pipeline themselves, the criterion is actually simple—once you need to track more than two or three counties, or once lead quality directly affects deal conversion, infrastructure investment starts to compound.

Address standardization and entity resolution

Address standardization (address normalization) is the process of parsing any written form of an address into a structured, comparable standard format, typically outputting separate fields like house number, street name, directional prefix, unit number, city, state, and ZIP+4, and mapping to authoritative coordinates (geocoding). When scraping across 14 counties, the same property may repeatedly appear as "123 Main St," "123 MAIN STREET," "123 Main St., Apt B," and so on. Without standardization, deduplication logic will treat the same target as multiple leads, both inflating lead volume and diluting scores. Complementary entity resolution further determines whether two records point to the same real-world object—the same property, the same owner. Together, these two determine the lead accuracy rate metric, and they're also why this article lists "address standardization confidence" as one of the scoring dimensions: low confidence means the record's identity attribution is questionable and should be down-weighted rather than directly accepted.

One-line Summary

A reusable approach to scaling a real estate wholesaling data pipeline from one county to 14 using a three-layer CrawlOS + Supabase + pg_cron architecture

Paragraph Summary

This article breaks down a production-grade real estate wholesaling data infrastructure: CrawlOS as the scraping and parsing abstraction layer, Supabase (Postgres) handling relational modeling across 17 tables, and pg_cron managing nightly staggered scheduling, covering 14 counties in the Houston metro area, scraping HCAD appraisal district data, and producing roughly 125 qualified leads per run. The author's core idea is to reframe the problem from "writing scrapers" to "designing data infrastructure": leads are defined as cross-table query results rather than individual records, so adjusting scoring rules requires no historical backfill; actors are deployed independently per county with retry and checkpoint-resume capability; and scrape success rate and lead accuracy rate are monitored as two independent metrics. The article also points out that the main challenges in cross-county scaling lie in field differences, anti-scraping changes, and address standardization polluting deduplication results.