#open-source web crawler
40 related articles
·3 min
Small Site Besieged by AI Crawlers: Under 100 Images, 266K Crawl Requests
A Piwigo gallery with under 100 images received 266K AI crawler requests in 24 hours. Analysis of why gallery CMS becomes a bot trap and how to defend against it.
Read more →
·3 min
MediaCrawler: Deep Dive into an Open-Source Crawler Framework Covering Seven Major Platforms
Deep analysis of MediaCrawler, a popular GitHub open-source project using Playwright browser automation for multi-platform crawling across Xiaohongshu, Douyin, Kuaishou, Bilibili, Weibo, Tieba, and Zhihu.
Read more →
·3 min
MediaCrawler: Deep Dive into an Open-Source Crawler Framework Covering Seven Major Platforms
Deep dive into the popular GitHub project MediaCrawler — a Playwright-based multi-platform crawler framework supporting data collection from Xiaohongshu, Douyin, Kuaishou, Bilibili, Weibo, Tieba, and Zhihu.
Read more →
·3 min
GitHub Daily · July 27: AI Agent Skill Ecosystem Explodes, Claude's Capability Boundaries Keep Expanding
GitHub Trending July 27: AI Agent Skills explode as claude-video, impeccable, and last30days-skill extend model capabilities without modifying models themselves.
Read more →
·3 min
Open-Source Automated News Briefing Generator: A Zero-Technical-Barrier AI Deployment Analysis
An open-source automated news briefing system. No coding needed—just let an AI read the project docs to complete the entire deployment. Six-stage pipeline, four-channel search covering 16+ platforms, smart classification and dedup, daily auto-push to Feishu, completely free.
Read more →
·3 min
Open-Source Newsletter Auto-Generation System: A No-Code AI Deployment Breakdown
An open-source newsletter auto-generation system requiring no coding—just have AI read the project docs to complete deployment. Six-stage pipeline, four-channel search covering 16+ platforms, smart classification & deduplication. Completely free.
Read more →
open-seo: A Deep Dive into the Free, O…
·2 min
open-seo: A Deep Dive into the Free, Open-Source Alternative to Semrush
open-seo is a trending open-source SEO tool on GitHub, positioned as a free alternative to Semrush and Ahrefs. This deep dive covers its tech stack, features, use cases, and limitations.
Read more →
·3 min
Harness Architecture in Practice: A Deep Dive into Enterprise-Level Agent Projects
A deep dive into Harness architecture in enterprise Agent projects, covering MCP protocol, sandbox isolation, multi-model scheduling, and ASGI deployment — key topics for LLM job interviews.
Read more →
Anti-AI Fonts: Adversarial Typography …
·3 min
Anti-AI Fonts: Adversarial Typography That Humans Can Read but Machines Can't
Anti-AI fonts use adversarial glyph design to exploit the perceptual gap between human and machine vision, blocking OCR and LLM scraping. A deep dive into the tech, limits, and data sovereignty implications.
Read more →
·3 min
Getting Started with LangChain: The Core Framework for LLM Application Development and Its Three Key Components
LangChain is the leading open-source framework for LLM application development, supporting GPT-4, GLM, and other mainstream models. This article dives into its three core concepts: Components, Chains, and Agents.
Read more →
·5 min
Prismata Analysis: How to Defend Web Agents Against Cross-Site Prompt Injection Attacks
Cross-site prompt injection is becoming the trickiest security threat for Web agents. This article analyzes the Prismata project's 'confining defense' approach—controlling injection's blast radius via context isolation, permission boundaries, and trust grading.
Read more →
·4 min
GPT-5.6 Sol vs Grok 4.5: Same $20 Subscription — Which One Is Actually Worth It?
GPT-5.6 Sol or Grok 4.5 — both at $20/month, but which delivers more value? We break down use cases, intelligence benchmarks, and ecosystem tradeoffs to help you decide.
Read more →
·8 min
OpenAI Merges Codex with ChatGPT: A Deep Dive into the Unified Desktop App Overhaul
OpenAI officially merges its coding agent Codex with ChatGPT into a unified desktop app, adding new coding workflows, a Chrome extension, a built-in browser, and GPT-5.6-powered Computer Use capabilities.
Read more →
·5 min
28 Enterprise-Grade AI Agent Projects: A Complete Guide That Even Beginners Can Deploy
A collection of 28 fully reproducible enterprise-grade AI Agent projects covering code debugging, financial analysis, customer service, and multi-agent collaboration—deployable even for beginners.
Read more →
·7 min
Free Mermaid Diagram Editor: An Essential Code-Based Diagramming Tool for Developers
Explore the core features and use cases of the free Mermaid Diagram Editor. Supporting flowcharts, sequence diagrams, Gantt charts and more, it follows the 'diagrams as code' philosophy to enable version-controlled technical documentation for developers and architects.
Read more →
·7 min
Learning Python from Scratch: A Complete Guide to the Three-Stage Learning Path
Want to learn Python from scratch but don't know where to begin? This article breaks down three stages—basic syntax, advanced mastery, and hands-on practice—with real projects in crawling, automation, and data analysis to help you build programming thinking.
Read more →
·6 min
Reverse-Engineering ChatGPT's Source Selection: What Network Traffic Analysis Reveals About AI Citations
By capturing and analyzing ChatGPT's real network traffic during web retrieval, this article reveals the underlying logic of AI source selection—covering the layered mechanisms of retrieval recall and content filtering, domain preference patterns, and practical insights for content creators and SEO professionals.
Read more →
·3 min
Harness Engineering in Practice: A Guide to Enterprise-Grade Multi-Agent System Architecture
A deep dive into Harness Engineering architecture: building an AI procurement assistant on ERP systems, covering multi-agent orchestration, MCP protocol, ASGI deployment, and sandbox isolation.
Read more →
Firecrawl Goes Free: No Sign-Up Requir…
·3 min
Firecrawl Goes Free: No Sign-Up Required to Start Scraping Web Data
Firecrawl is now free with no sign-up required. Learn about its evolution from open source to zero-friction access and why web data matters for AI Agents.
Read more →
Web Monitoring API Launches: Giving AI…
·3 min
Web Monitoring API Launches: Giving AI Agents Real-Time Awareness of Changes Across the Web
A new Web Monitoring API lets AI agents proactively detect changes across the entire web. Supports event-driven notifications for competitive intel, e-commerce, finance, and more. Free to integrate.
Read more →