ClearMesh: A Git Version Control Platform That Manages Large Files Like Code
ClearMesh brings Git-style version control to large files with chunk deduplication, encryption, and streaming mounts.
ClearMesh is a large file version control tool for AI, VFX, and research teams, addressing the limitations of traditional Git and Git LFS. It uses chunked storage for cross-version deduplication, supports S3/R2 backends, provides client-side zero-knowledge encryption, and offers FUSE-based read-only streaming mounts with a near-native Git experience — offering clear advantages over Git LFS and DVC in deduplication, encryption, and streaming access.
Overview
Teams in AI, VFX, scientific research, and engineering deal with massive datasets, model files, and binary assets every day. Traditional Git has limited support for large files — Git's content-addressable object storage is highly efficient for diffing and compressing text files, but has fundamental limitations with large binary files: every commit stores the entire file in the object database, causing repository size to grow linearly with the number of versions. Git LFS (Large File Storage), released by GitHub in 2015, mitigates this by replacing large files with pointer files and storing the actual content on a separate server. However, it essentially just "moves" the files without providing block-level deduplication, and its support for branching, merging, and offline work is weak. Meanwhile, existing file storage solutions lack version control capabilities. ClearMesh was built to solve exactly this pain point — it brings Git workflows to large file management, enabling teams to manage data assets just like they manage code.
Core Features and Design Philosophy
Git-Style Version Control for Large Files
ClearMesh provides a complete Git-like experience for large files, supporting core operations such as commit, push, clone, sync, and branch. AI teams can version-control training datasets, VFX teams can track every change to media assets, and engineering teams can manage the iteration history of CAD exports.
The biggest advantage of this design is the minimal learning curve — any developer familiar with Git can get started quickly while gaining professional-grade management capabilities for large files.
Smart Chunked Storage: Saving Space and Cost
ClearMesh's underlying storage uses a chunking mechanism, where files are split into multiple chunks stored in S3/R2-compatible Vault storage. Chunked storage is at the core of modern data deduplication: the system splits files into chunks of fixed or variable size and computes a hash (e.g., SHA-256) for each chunk as a unique identifier. When a new file version is uploaded, the system only uploads chunks whose hashes have changed — unchanged chunks reuse existing storage. This is philosophically aligned with Git's object storage approach but operates at a finer granularity. Variable-Length Chunking (e.g., using the Rabin fingerprint algorithm) can more intelligently identify change boundaries within files, achieving higher deduplication rates for scenarios like inserting or deleting records in a dataset. This architecture delivers two key advantages:
- Incremental storage and deduplication: Unchanged chunks can be reused across versions, dramatically saving storage space. For large datasets that iterate frequently but only change partially each time, storage costs can be significantly reduced.
- Flexible storage backends: Supports object storage services like Amazon S3 and Cloudflare R2, allowing teams to choose the best option based on budget and performance requirements.
Client-Side Encryption for Data Security
ClearMesh offers optional client-side encryption, where data is encrypted before upload. Client-Side Encryption means the storage provider holds ciphertext rather than plaintext — even if the provider suffers a data breach or insider abuse, attackers cannot access meaningful data. This architecture is commonly called "Zero-Knowledge" mode, which is fundamentally different from Server-Side Encryption, where keys are managed by the provider and the provider can theoretically decrypt the data. For teams subject to compliance requirements like GDPR and HIPAA, client-side encryption is an important technical measure for meeting data sovereignty requirements, ensuring data remains secure even if the storage layer is compromised. This is especially important for teams handling sensitive research data or proprietary commercial assets.
Read-Only Mounting and Streaming Access
ClearMesh supports mounting repositories as read-only on the local filesystem, allowing tools to stream files through standard file paths. This feature is typically implemented using FUSE (Filesystem in Userspace) — FUSE allows developers to write filesystem drivers in userspace, mapping remote storage to local directories without kernel modifications. When applications read data through standard file paths, the FUSE driver fetches the corresponding data chunks from object storage on demand, enabling true streaming access. This mechanism is particularly critical for machine learning training scenarios: training frameworks (such as PyTorch DataLoader) can read data directly from the mounted path without being aware of the underlying storage details, and can start training without waiting for the full dataset to download, significantly reducing experiment startup time. The practical benefits of this feature include:
- Start working without downloading the entire dataset locally
- Existing toolchains can access versioned files without any modifications
- Multiple collaborators share the same data source, avoiding bandwidth waste from redundant downloads
Use Cases
Dataset Management for AI/ML Teams
For machine learning teams, ClearMesh can manage versions of training datasets and model weight files. As models iterate, teams can clearly trace the data used and models produced by each version, achieving full experiment reproducibility.
Asset Collaboration for VFX and Creative Teams
Media assets handled by visual effects and creative teams are typically large and frequently iterated. ClearMesh's branching feature allows different artists to work independently on their own branches and merge results at the end, avoiding file conflicts and overwrites.
Research Data and Data Engineering
Research teams can version-control experimental data, and data engineers can track output changes at every stage of data pipelines, ensuring complete and traceable data lineage.
ClearMesh vs. Git LFS and DVC
DVC (Data Version Control) was open-sourced by Iterative.ai in 2017 and is one of the earliest tools in the MLOps space to systematically address data and model versioning. It builds a metadata management layer on top of Git, storing actual data file contents remotely (S3, GCS, etc.) while keeping only lightweight .dvc pointer files in the Git repository. DVC also introduces the concept of Pipelines, which can track each stage of data processing and its dependencies. However, DVC has a relatively steep learning curve, its command system differs from Git, and it also lacks block-level deduplication — leaving room for differentiation by newcomers like ClearMesh. ClearMesh positions itself as an alternative to tools like Git LFS and DVC:
| Feature | Git LFS | DVC | ClearMesh |
|---|---|---|---|
| Native large file support | Limited | Good | Natively designed |
| Chunk-level deduplication | Not supported | Not supported | Supported |
| User experience | Depends on Git | Separate commands | Near-native Git |
| Streaming mount | Not supported | Not supported | Built-in |
| Client-side encryption | Not supported | Not supported | Optional |
ClearMesh is currently free to use. As a YC application project, its future business model will likely revolve around enterprise features and storage usage.
Summary
ClearMesh fills an important gap in large file version control. It combines Git's streamlined workflows with a chunked storage architecture purpose-built for large files, augmented by client-side zero-knowledge encryption and FUSE-based streaming mount capabilities, providing a practical solution for teams handling massive binary assets. If your team is struggling with dataset management, model version tracking, or large asset collaboration, ClearMesh is worth exploring.
Key Takeaways
- ClearMesh provides Git-style version control operations for large files (datasets, models, binary folders)
- Uses a chunked storage mechanism where unchanged chunks are reused across versions, significantly saving storage space
- Supports S3/R2-compatible storage backends with optional client-side encryption for data security
- Supports read-only mounting, allowing tools to stream versioned files through standard file paths
- Suited for large file collaboration scenarios across AI, VFX, research, and engineering teams
Related articles
Product ReviewsThe Programmer's Desk Setup Guide: Building a Workspace That Feels Like Home
Discover how programmers build productive, comfortable workspaces. From multi-monitor setups to ergonomic design, explore the desk philosophy that drives focus and flow.
Product ReviewsQoder vs Cursor Real-World Comparison: Which $20/Month AI IDE Is Better?
Hands-on comparison of Qoder vs Cursor AI IDEs: Agent autonomy, human interaction count, and architecture decisions. Qoder needed only 2 interactions vs Cursor's 8.
Product ReviewsCursor Cloud Agent Demo: Eliminating Bottlenecks Across the Entire Software Development Lifecycle
Deep analysis of Cursor's Cloud Agent demo showing how cloud VMs, automated test artifacts, and a full-chain control plane systematically eliminate human bottlenecks across the software development lifecycle.