Replacing File Subscriptions with OpenSharing: A New Paradigm for Data Sharing

Many data-sharing architectures start with a familiar pattern: the provider exports data and distributes it to consumers through file-based sources. This "copy-based" sharing looks simple, but it brings data latency, storage redundancy, and consistency problems. OpenSharing proposes a "read, don't copy" approach. This article is the second part of its practical guide and focuses on the concrete implementation path.

Why Replace File Subscriptions?

In the traditional file subscription model, providers periodically generate data files (such as CSV or JSON), and consumers download and import them into their own systems. This model still works when data volume is small and real-time requirements are low, but as data scale grows and more collaborators join, the problems gradually surface:

• Data latency: File generation and distribution happen on a cycle, so consumers receive a "snapshot" that cannot reflect the latest state.
• Storage and compute redundancy: The same data is copied multiple times, consuming large amounts of storage, and every consumer must repeatedly parse and process it.
• Difficult consistency maintenance: Multiple replicas easily drift apart, and version management becomes complex.

The core idea of OpenSharing is to let consumers directly "read" the provider's data instead of "copying" it. This is similar to moving from "downloading a document" to "collaborative online editing", where there is always a single authoritative source of data.

OpenSharing Implementation Architecture

Implementing OpenSharing requires focusing on three key components: the data provider, the sharing contract, and the consumer connector.

Data Provider: Expose Readable Interfaces

The provider no longer exports files. Instead, it exposes datasets through standardized APIs or protocols. This can be a RESTful interface, a GraphQL endpoint, or shared storage based on open table formats (such as Delta Lake or Iceberg). The key point is that the provider retains full control over the data and can update it at any time, while consumers read on demand.

Open table formats are a class of specifications that define table abstractions and transactional metadata on top of data lakes. Typical examples include Delta Lake, Apache Iceberg, and Apache Hudi. They organize a set of files in object storage into a "table" with schema, partitions, and version history, and implement ACID semantics, time-travel queries, and incremental reads through a metadata layer (such as a transaction log or manifest files). For OpenSharing, the value of open table formats is that the provider does not need to export files, consumers can read a consistent data view by snapshot or incremental change, and multiple engines (Spark, Trino, Flink, etc.) can share the same authoritative data.

Sharing Contract: Define Access Rules

The sharing contract is a metadata description agreed upon by both parties, including data schema, access permissions, terms of use, and update frequency. It is similar to an API contract but focuses more on data semantics. The contract can exist as a declarative configuration, ensuring both parties have a consistent understanding of the data and reducing integration friction.

Consumer Connector: Pull on Demand

Consumers read data through a connector, which handles authentication, protocol conversion, and caching strategies. Unlike file subscriptions, the connector does not store a complete copy. Instead, it pulls only the data fragments needed for a query. This requires the connector to have some query pushdown capability to reduce network traffic.

Query pushdown means that the connector translates the consumer's filter, projection, aggregation, and other operations as much as possible into native queries that the provider interface can understand, so the provider executes them locally and returns only the result set, instead of pulling the full dataset to the consumer for computation. The benefit is a significant reduction in network transmission and local compute overhead, especially for wide tables or highly selective queries. Implementation requires the connector to have a capability negotiation mechanism to identify which pushdown operators the provider supports; parts that cannot be pushed down fall back to local execution. When designing, pay attention to pushdown boundaries and result correctness to avoid data drift caused by semantic differences.

Implementation Steps and Considerations

Migrating from file subscriptions to OpenSharing is not achieved overnight. It is recommended to proceed in phases:

1. Assess existing data flows: Identify which datasets are suitable for real-time reads and which can still use file distribution (such as historical archive data).
2. Design sharing contracts: Work with consumers to define data schemas, access frequency, and SLAs to ensure both sides have aligned expectations.
3. Build provider interfaces: Prioritize technology stacks familiar to the team, and gradually introduce open table formats to support incremental reads.
4. Develop consumer connectors: Start with simple query scenarios, and gradually add caching and query optimization logic.
5. Monitor and iterate: Establish data quality monitoring, track read latency and failure rates, and continuously optimize contracts and interfaces.

During implementation, note the following: network latency may become a new bottleneck, so caching strategies must be designed properly; permission management must be fine-grained to the field level to avoid over-exposure; contract changes require version management to prevent unexpected consumer disruptions.

Field-level permission control requires the permission model to be refined to the column or even row/cell granularity, rather than only table-level authorization. Common practices include: marking each field's sensitivity level and visible roles in the sharing contract; the provider implementing column masking, row filtering, or dynamic views at the query layer; the connector carrying identity and purpose declarations in requests, with the provider's policy engine (such as attribute-based access control, ABAC) making decisions. Field-level permissions can reduce the risk of over-exposure, but they also increase contract complexity and query overhead. A balance must be struck between governance cost and security benefit, and this should be included in version management.

Benefits and Trade-offs

After adopting OpenSharing, data freshness improves significantly, and consumers can obtain the latest data almost in real time. Storage redundancy is greatly reduced because there is no need to maintain separate copies for each consumer. Consistency is also easier to guarantee because there is a single authoritative source.

However, this model also brings new challenges: the availability of the provider interface directly affects all consumers, so higher operational assurance is required; consumer query performance depends on the network and provider response speed, and latency-sensitive scenarios may require additional optimization. In addition, inter-organizational trust and contract governance require effort to establish.

Overall, OpenSharing represents the evolution of data sharing from "file movement" to "service-based reads". For teams pursuing real-time performance and consistency, this is a path worth exploring.

One-line Summary

OpenSharing restructures data sharing with "read, don't copy", offering a three-component architecture and a phased migration path.

Paragraph Summary

This article is the second part of the OpenSharing practical guide, focusing on the concrete implementation of migrating from traditional file subscriptions to the "read, don't copy" model. It first analyzes the inherent flaws of file subscriptions in data latency, storage redundancy, and consistency, then proposes a three-component architecture consisting of the data provider, sharing contract, and consumer connector: the provider exposes data through APIs or open table formats and retains control, the contract declares schemas, permissions, and SLAs, and the connector pulls on demand and supports query pushdown. It then presents a five-step phased implementation path and considerations such as network latency, field-level permissions, and contract versioning, and finally weighs the benefits and new challenges, concluding that this is the evolution direction of data sharing from file movement to service-based reads.