Beyond SFTP and API Polling: A New Data Sharing Paradigm with OpenSharing

Why Your Data Pipeline Needs a Paradigm Shift

If you're pulling data from customers via SFTP or APIs, you've probably asked yourself repeatedly: why does onboarding every new customer mean building another file transfer channel from scratch? Why does a single API version change force the entire data pipeline to be rebuilt? According to a technical article from an RSS source, Read, don't copy: replacing file and API feeds with OpenSharing - Part 1 (The Why), traditional file delivery and API polling models are becoming bottlenecks for data collaboration, and an approach called OpenSharing aims to fundamentally change the status quo.

The article's core argument isn't complicated: the essence of data sharing should be "reading," not "copying." Around this idea, the author offers a systematic critique of the two mainstream approaches—SFTP and APIs—and makes the case for OpenSharing as an alternative.

SFTP and APIs: Seemingly Mature, Actually Costly

The Hidden Burden of SFTP

SFTP has long been the default option for business-to-business file exchange, but its convenience masks substantial operational costs. Every new customer onboarding means coordinating credentials, configuring directories, monitoring file arrivals, and handling format discrepancies. Once a file arrives, the receiver still has to parse it, validate it, and load it into a database—and the entire chain lacks a unified data contract. Worse, SFTP transfers are typically "snapshots"—copies of data at a point in time—and subsequent changes can't be synced automatically, causing both parties' understanding of the data state to gradually drift apart.

SFTP (SSH File Transfer Protocol) is not a standalone protocol but a subsystem running on top of SSH, typically reusing port 22 and thus inheriting SSH's key-based authentication and encrypted channel by nature. Its purpose is "secure file movement," not "data exchange protocol": the protocol itself carries no schema, primary keys, change timestamps, or delete markers, nor does it define how the receiver confirms a record has been processed. In practice, idempotency, deduplication, and resumable transfers are cobbled together through filename conventions (such as date suffixes) or extra control files—exactly the technical root of what the article calls the "lack of a unified data contract." Meanwhile, SFTP batch processing typically runs during overnight windows, naturally introducing T+1 latency.

The Versioning Trap of APIs

APIs look more modern, but the article points out that the API polling model doesn't solve the fundamental problem either. APIs require consumers to actively poll or subscribe, and interface design, authentication methods, pagination strategies, and rate limits are all defined by the provider—consumers can only adapt passively. Once a provider upgrades the API version or adjusts fields, consumer code has to change accordingly. This "copy locally, then process" model forces data to be shuttled repeatedly between multiple systems, increasing latency and amplifying the risk of inconsistency.

REST API versioning is typically indicated through URL paths (/v1/, /v2/) or request headers, but semantic versioning doesn't enforce backward compatibility—when providers delete or rename fields, consumers often only discover the errors at runtime. Pagination and incremental sync are another hidden cost: most APIs offer only offset-based pagination and timestamp filters like updated_since, and the latter can miss records when clock skew or out-of-order writes occur. Rate limits force consumers to design backoff retries and local caching—and once a cache is in place, you're back to the "copy" model, with drift between the replica and the source that's hard to detect systematically.

The Core Idea of OpenSharing: Read, Don't Copy

The "Read, don't copy" in the article's title captures OpenSharing's design philosophy. In traditional models, data is copied from provider to consumer, and the copying process itself is the source of problems: replicas go stale, formats drift, and permissions spiral out of control. OpenSharing aims to let consumers "read" source data directly, rather than making a copy first and using that.

The immediate benefit of this approach is guaranteed data freshness and consistency—consumers see the provider's current state, not a snapshot from some point in time. At the same time, since there's no longer a need to build a separate transfer channel for each customer, onboarding costs can be significantly reduced. The article positions OpenSharing as a wholesale replacement for both file delivery and API polling models, not just another wrapper layer on top of them.

In technical terms, "direct reading" typically corresponds to the combination of open table formats and sharing protocols. Open sharing protocols such as Delta Sharing let providers grant consumers access to tables or partitions, and consumers use their familiar engines (Spark, Pandas, Trino, etc.) to read Parquet data files and metadata logs through short-lived tokens—without moving the entire dataset into their own storage. The "read" here may still involve transferring bytes over the network, but semantically the consumer is accessing the same table the provider continuously maintains, not a one-off snapshot. This requires providers to maintain a storage layout that can be read externally and fine-grained authorization—precisely why the article emphasizes that permission models need to be redesigned.

This Is Not Just Another API Gateway

It's important to distinguish that OpenSharing is not simply an API wrapped in a unified gateway. The article emphasizes a shift in sharing semantics: from "I send you a file" or "you call my interface" to "you read my data directly." This shift involves redesigning permission models, data contracts, and access control—not optimizing the transport layer. For data teams long plagued by SFTP and custom API integrations, this represents an architectural direction worth watching.

Who Should Pay Attention to OpenSharing

If your team maintains a large number of customer data ingestion pipelines, or if onboarding each new customer takes days or even weeks of integration work, then the problems OpenSharing discusses are exactly your pain points. As the first part of a series, the article focuses on the "why"—the costs and limitations of existing models—and subsequent parts are expected to cover concrete implementation approaches. For data engineers and platform architects, understanding this line of reasoning helps make more informed judgments when evaluating building pipelines in-house versus adopting a new sharing model.

OpenSharing is still at the conceptual stage, and its ecosystem maturity and practical feasibility remain to be seen. But the problems the article raises—the latency, inconsistency, and onboarding costs caused by copying—are real and deserve serious consideration in architectural decisions.

One-line Summary

The article argues for replacing SFTP and API "copy" models with OpenSharing's "direct read" approach.

Paragraph Summary

Centered on the core thesis "Read, don't copy," the article systematically critiques the two mainstream approaches to enterprise data exchange. SFTP seems convenient but accumulates substantial operational costs in credential coordination, directory configuration, format discrepancies, and the lack of a data contract—and what it transfers is a snapshot whose changes can't be synced automatically. API polling falls into a versioning trap: interface design, authentication, pagination, and rate limiting are all defined by the provider, forcing consumers to adapt passively and rewrite code whenever fields or versions change—while "copy locally, then process" brings latency and inconsistency. OpenSharing aims to let consumers read source data directly, ensuring freshness and consistency while lowering onboarding costs. The article stresses that it's not an API-gateway-style wrapper but a shift in sharing semantics at the level of permission models and data contracts, and as the first part of a series, it focuses on making the "why" case.