Painterly: Turning Photos into Oil Paintings with Algorithms — No Generative AI Required

A Counter-Current Image Stylization Tool

In an era where generative AI dominates the image space, nearly every "turn photos into art" tool relies on Stable Diffusion, Midjourney, or similar diffusion models. Yet a tool called Painterly is swimming against the tide — explicitly claiming to convert ordinary photos into digital painting-style artwork without using any generative AI.

This project debuted on Hacker News as a Show HN post, representing a fundamentally different technical approach to image stylization. At a time when the entire industry is chasing larger models and more powerful compute, Painterly reminds us that there's still a more "classical" algorithmic path to transforming photos into paintings.

What a Non-Generative Photo-to-Painting Approach Actually Means

Algorithm-Driven, Not Model-Driven

By "no generative AI," Painterly means it doesn't rely on neural networks trained on massive image datasets to "imagine" and repaint pictures. Instead, it uses traditional computer graphics and image processing algorithms to simulate painting brushstrokes.

This class of techniques is known as NPR (Non-Photorealistic Rendering) or Stroke-Based Rendering — a research branch within computer graphics with over 30 years of history. Salisbury et al.'s pen-and-ink rendering paper at SIGGRAPH in the 1990s, and Hertzmann's classic stroke-based rendering algorithm from 1998, laid the theoretical foundation for this field. The central challenge: how to make computers understand the perceptual decisions human painters make — where to use bold, sweeping strokes, and where to apply fine, delicate detail. The core approach involves:

• Analyzing the image's edges, color regions, lighting, and texture (typically combining Canny edge detection, saliency analysis, and color quantization)
• Generating simulated brush strokes based on this information
• Layering strokes in a specific order, direction, and scale to produce visual effects resembling oil painting, watercolor, or sketches

This methodology has decades of research behind it in the digital art world — long before deep learning emerged, academia had already published extensive work on stroke-based rendering and image abstraction.

The Fundamental Difference from Generative AI Approaches

Generative AI image stylization has gone through two major generations. The first was Neural Style Transfer, proposed by Gatys et al. in 2015, which uses Gram matrices from convolutional neural networks to match feature distributions between a content image and a style image. The second is diffusion models like Stable Diffusion, which generate images by iteratively denoising from noise and can interpret text instructions. Both approaches share a reliance on large training datasets, and their inference processes are essentially "black boxes" — it's difficult to intuitively explain mathematically why a model produces any given output.

By contrast, algorithm-driven photo-to-painting conversion is faithfully redrawing the original image — preserving the original photo's composition, content, and color relationships, while replacing pixel-level photorealism with painterly brushstroke representation. Every transformation step has an explicit mathematical correspondence and is fully traceable and interpretable.

Why This Still Matters in the Age of AI

Controllable, Predictable, and Explainable

One of generative AI's biggest pain points is unpredictability — the same input can produce wildly different outputs, with no easy way to make precise adjustments. Traditional algorithmic approaches, by contrast, are deterministic and explainable at every step. Users can precisely tune parameters like brush size, density, and color saturation to get stable, consistent results.

Lightweight and Privacy-Preserving

Algorithm-driven image stylization tools typically don't require large model files or GPU compute — they can run locally in a browser or on ordinary hardware. A key enabling technology here is WebAssembly (WASM) — a binary instruction format that runs at near-native speed in modern browsers, allowing developers to compile high-performance image processing code written in C/C++ or Rust directly into a browser-executable format with no server-side support needed. As this technology has matured, computationally intensive image algorithms that once required server infrastructure can now run entirely on the user's device, delivering two direct benefits:

• Minimal performance overhead: No need to upload images to cloud servers for inference, eliminating network latency
• Privacy-friendly: Photos can be processed entirely locally, fundamentally preventing image data from being uploaded or used for training

This also aligns with the growing Local-First software philosophy — the idea that data should be stored and processed on the user's own device whenever possible.

Avoiding Copyright and Training Data Controversies

Since the generative AI image tool explosion of 2022, copyright disputes have continued to escalate. Cases like Getty Images vs. Stability AI and artists' class-action lawsuits against Midjourney share a central question: were the hundreds of millions of images used to train these models authorized by their original creators? The U.S. Copyright Office clarified in 2023 that purely AI-generated images are not eligible for copyright protection, while the boundaries for human-AI collaborative works remain murky.

Painterly and tools like it involve no training data whatsoever — their output is based entirely on the user's own original images, with clear copyright attribution that remains with the original photographer. This sidesteps the sensitive question of "whether AI infringes on artists' rights" entirely, which is a meaningful advantage for compliance-conscious creators and businesses.

The Strengths and Limitations of Painterly's Technical Approach

Where It Shines

Tools like Painterly demonstrate unique value in the following scenarios:

• Situations requiring batch processing of large numbers of images with consistent results
• Applications with strict requirements for faithfulness to the original content
• Individual users who prioritize privacy protection and local processing
• Commercial projects looking to avoid the copyright risks of generative AI

Real-World Limitations

That said, non-generative approaches do have a ceiling. Compared to style transfer models trained at scale, purely algorithmic photo-to-painting solutions often struggle to replicate complex artistic styles — like the distinctive brushwork of a specific painter. Their output tends toward "generic painting filter effects" rather than highly personalized artistic reinterpretation. Additionally, stroke-based rendering quality is highly sensitive to parameter tuning, which presents a learning curve for general users.

Closing Thoughts: The Value of Technical Diversity

Painterly's existence is, at its core, a gentle pushback against the idea that AI is the answer to everything. At a time when generative AI dominates nearly every image-related conversation, it demonstrates that traditional NPR algorithms still have irreplaceable value — especially in scenarios where controllability, privacy, and copyright compliance are core requirements.

For developers and creators, this is an important reminder: when solving a specific problem, the newest and most hyped technology is not always the best solution. Understanding the fundamental differences between technical approaches — whether it's the creative reimagination of diffusion models or the faithful simulation of NPR algorithms; whether it's the raw power of cloud inference or the local-first processing enabled by WebAssembly — and choosing based on actual needs, is a sign of mature engineering judgment. Painterly may never go viral, but the "return to algorithmic fundamentals" mindset it represents is a remarkably clear-headed perspective in an era increasingly dominated by large models.