End Duplicate Uploads with One File: An Adobe Stock Batch Publishing Tracker

Title

End Duplicate Uploads with One File: An Adobe Stock Batch Publishing Tracker

Meta Description

Duplicate submissions caused by re-running scripts during Adobe Stock batch uploads are a high-risk operation. This article introduces a developer's complete solution and practical habits for fixing duplicate uploads with a single-file Python upload tracker plus a deterministic metadata generator, with zero dependencies.

Article Content

When batch-submitting work to asset platforms like Adobe Stock, duplicate uploads caused by re-running scripts are a high-risk operation that can get a Contributor account flagged. One developer shared his lightweight fix: a single-file upload tracker that depends only on the Python standard library, plus a companion deterministic metadata generation script, to fundamentally eliminate "memoryless re-runs."

How Duplicate Uploads Happen

The root of the problem is that in the submission workflow, the two steps "generate metadata" and "submit" are not atomic operations. Adobe Stock's CSV template requires a strict five-column header, Vecteezy needs four columns, and Dreamstime needs fifteen. Any header error causes the channel to immediately reject the files, and the human instinct is to fix the CSV and re-run the entire batch.

The script has no memory. It will happily regenerate entries for files that have already been submitted, because the fact of "already submitted" exists only in the Contributor backend, not on the local disk. Partial failure makes things worse: if a batch of 40 files is interrupted at file 30, you neither want to re-upload the first 30 nor have a ledger telling you where it stopped, so you can only re-upload everything or manually audit the backend—and the latter is exactly the tedious work the pipeline was supposed to eliminate. This developer discovered late one Tuesday night that the same batch of vector assets had appeared twice in the review queue, and within a week 40 files had been submitted twice.

Non-atomic operations are the key concept here. In databases and distributed systems, an atomic operation refers to an indivisible sequence of operations that either all succeed or all fail. The submission process described in the article consists of two steps, "generate metadata" and "submit files," with no transactional guarantee between them: CSV generation may succeed, but submission may partially fail or be interrupted, and the system will not roll back the parts already submitted. This "partially complete" state cannot be perceived during a re-run, so the script regenerates entries for all files (including those already submitted), causing duplicate uploads. Common patterns for solving non-atomic problems include: introducing idempotency keys (such as file hashes) so that duplicate submissions are ignored by the platform, or using a local state machine to record each file's stage (generated, submitting, completed), ensuring that completed stages are skipped on re-run. The tracker in the article is essentially a lightweight state machine.

The Tracker Is Just One File, with Zero Dependencies

The solution is embarrassingly small. upload_tracker.py is a pure-standard-library Python script that requires no pip install, uses a plain-text ledger to record completion status, and answers three questions:

python upload_tracker.py pending items.txt   # what's left to do
python upload_tracker.py done "asset-001.jpg"
python upload_tracker.py status

The workflow that replaces the "Tuesday disaster" is: before any submission run, generate items.txt from the folder, run pending first, submit only the files it shows, and mark each one done as it is sent. If the run is interrupted at file 30, re-running pending will show only the last 10 files, and only those 10. Re-running can no longer cause duplicate uploads, because what decides what gets sent is the ledger, not memory.

The zero-dependency aspect is worth emphasizing. The author's entire pipeline runs on a server with 1 vCPU, 2 GB of memory, and no GPU, and his toolset has only two scripts that need third-party packages (Pillow for covers, reportlab for PDF generation). The metadata builder and the tracker only need Python 3.9+, which means they can run in any environment, including this small machine, with no need to argue about virtual environments.

Metadata Is the Other Half of the Same Problem

The tracker is only useful if the CSV is correct the first time. make_metadata.py writes files with exactly the headers each channel requires, no more and no less:

python make_metadata.py ./my-assets --platform adobe      --out metadata.csv   --ext .jpg
python make_metadata.py ./my-assets --platform vecteezy   --out vecteezy.csv   --ext .jpg
python make_metadata.py ./my-assets --platform dreamstime --out dreamstime.csv --ext .jpg

If a titles.json is provided, the script can read real titles and keywords from it; when it is not provided, filenames are converted into title format as titles, which is good enough for a first pass but should be replaced before submission. CSV-stage rejections are exactly the source of the urge to re-run, so fixing the header problem and fixing the duplicate problem are essentially the same fix: make the pipeline deterministic, and you will never have the urge to "just run it again and see."

Deterministic metadata generation means that given the same inputs (file list, titles, keywords), the script outputs exactly the same CSV content every time (including row order, column order, and escaping format). This eliminates the possibility that "re-running changes the CSV," thereby avoiding platform rejections triggered by header or format differences. Key points for achieving deterministic output include: fixing the file traversal order (for example, sorting by filename), fixing the date-time format (to avoid the local time zone or current time influencing output), and fixing the column order and delimiters. In the article, make_metadata.py hardcodes each platform's headers through the --platform parameter, which is precisely the embodiment of determinism. Determinism is a prerequisite for idempotency: only when the output is reproducible can you safely re-run without side effects.

Three Habits for Staying Safe

The author sticks to three rules for every batch:

• Run only one process at a time. Never parallelize image tasks on a single-core machine. Serial execution looks slower per item, but it is faster overall, because nothing dies.
• Use gc.collect() between projects to release memory, and measure peak RSS rather than average using resource.getrusage(RUSAGE_SELF).ru_maxrss. Average memory tells you nothing about limits.
• Have each subprocess handle only one file during re-rendering steps, so memory is truly returned to the operating system between assets.

These lessons come from running a full pipeline of about 19,000 lines, packaging 39 products, and producing about 856 MB of finished output. The author says that if everything else were lost, the tracker is the first component he would rebuild.

Peak RSS (Resident Set Size) is a metric of the physical memory actually occupied by a process, and ru_maxrss records the maximum resident set size reached during the process's lifetime. Compared with average memory, peak RSS better reflects memory bottlenecks: on a single-core server with 2 GB of memory, once the peak exceeds available memory, the process will be terminated by the OOM killer, interrupting the batch. The author emphasizes measuring peak rather than average because averages mask transient memory spikes. gc.collect() manually triggers garbage collection, which can promptly release memory occupied by circular references or large objects; processing file by file in subprocesses lets the operating system reclaim all memory when each subprocess exits, avoiding the accumulation of memory leaks. Together, these habits ensure that long batch tasks run stably in resource-constrained environments.

Implications for Automated Publishing Workflows

The value of this case goes beyond Adobe Stock itself. Any multi-platform content distribution—stock asset libraries, e-commerce listings, podcast distribution—has the same non-atomic submission problem. The core idea can be summarized in three points: pull the "completed" state from the platform backend back to the local disk; use deterministic output to eliminate the motivation to re-run; and in resource-constrained environments, prefer zero-dependency, serial execution solutions. For creators using AI tools to batch-generate and distribute content, this approach is especially worth borrowing: generation can be fast, but submission must have a ledger.

The author packaged the tracker, the metadata builder, and two other scripts (a Markdown-to-PDF converter and a product packager that packages files and renders 1280x720 covers and 600x600 thumbnails) into the Pipeline Starter Kit, which shares its origin with his vector prompt pack and runs on hardware costing just a few dollars a month.

One-line Summary

Eliminate duplicate uploads in batch submissions with a zero-dependency tracker and deterministic metadata scripts.

Paragraph Summary

The article addresses the problem of duplicate uploads caused by re-running scripts when batch-submitting work to asset platforms like Adobe Stock, and proposes a lightweight fix. The author points out that the root cause of duplicate uploads is that "generate metadata" and "submit" are non-atomic: CSV header errors trigger channel rejections, and after fixing them the whole batch is re-run, while the script has no local memory, causing already-submitted files to be processed again. The solution is a single-file tracker, upload_tracker.py, that depends only on the Python standard library and maintains a plain-text ledger through the three commands pending/done/status to decide which files are actually submitted each time; the companion make_metadata.py generates deterministic CSVs with exactly the headers each platform requires, eliminating the urge to re-run at the source. The author also summarizes operational habits on a single-core server: serial execution, releasing memory with gc.collect(), measuring peak RSS, and processing files one at a time in subprocesses. This approach can be transferred to any multi-platform batch distribution scenario.