Automate Job Search with AI Agents and Residential Proxies for Precise Matching

The most time-consuming part of job hunting is often not the interview, but filtering out the truly matching listings from a sea of recruitment information. A developer built an AI Agent using MCP (Model Context Protocol), combined with residential proxies to scrape thousands of DevOps positions, and then had Claude score and rank each one against a resume. The entire process does not auto-submit applications—it only performs "precise filtering," handing repetitive work to the machine while keeping decision-making in human hands.

Why Use Residential Proxies to Scrape Jobs?

Scraping recruitment sites directly from a local IP quickly reveals problems. In the video demonstration, scraping without a proxy was rejected by 362 sites, returning "server busy" or "robot page" instead of job data. Many recruitment sites detect crawler behavior, and some have regional restrictions, only accessible from specific countries.

The solution is a residential proxy. Unlike data center proxies, residential proxy IPs come from real home broadband, with behavior patterns consistent with ordinary users, making them less likely to trigger risk controls. The author chose DataImpulse, which charges by traffic—$1/GB, pay-as-you-go, no subscription—and purchased 5GB in the demo. Configuration only requires filling in the proxy service's username, password, host, and port into the project's .env file, then running a test script to verify connectivity.

The test results showed: direct IP located in Canada, while after proxying the exit IP became New York, USA, confirming the proxy was active. The project also configured rotating and sticky IPs covering the US, UK, Germany, India, and other countries, laying the groundwork for subsequent cross-regional scraping.

Difference Between Residential and Data Center Proxies: Although data center proxy IPs are also held in bulk by service providers, their network segments are concentrated and ownership is clear. Recruitment sites' risk control systems can easily flag them as data center traffic via ASN (Autonomous System Number) or IP reputation databases and block them outright. Residential proxies instead borrow end users' home broadband IPs through partner networks to forward requests. These IPs are naturally tied to real ISPs and geographic regions, appearing no different from ordinary visitors to target sites. The trade-off is higher bandwidth costs, slightly higher latency, and some providers impose limits on concurrency and session duration. The pay-per-traffic model chosen by the author suits scraping scenarios with "many requests, small page sizes"—pay for what you use, no budget wasted on subscription cycles.

Comparative Experiment: Without vs. With Proxy

The author conducted a intuitive controlled experiment. When scraping without a proxy, only some of 638 recruitment pages succeeded, 362 requests were rejected, and about 4,000 job listings were obtained, of which only 9 explicitly offered visa sponsorship. After using a residential proxy for the same scraping task, all 1,000 recruitment pages returned successfully, using 37 different residential IPs, with zero errors and zero rejections. Over 5,200 job listings were scraped, with all requests returning 200 status codes.

This data shows that proxies don't just "bypass blocks"—they directly affect data completeness. Rejected requests mean you simply never see those opportunities. Of course, the author emphasizes this is "ethical" usage: information retrieval only, no large-scale automated applications.

Project Structure: From profile.md to MCP Server

The entire project revolves around several core files:

• sources: Lists recruitment sources to scrape, including job boards with public APIs like Greenhouse, Lever, Ashby, as well as large sites like Amazon Jobs.
• MCP server logic: Defines instructions, message log levels, and available tools when scraping data through the proxy.
• Geo.py: Provides geographic reference data for currencies, IPs, URLs, etc.
• profile.md: The job seeker's "digital resume," specifying target roles (e.g., DevOps Engineer, Platform Engineer, Cloud Engineer, Solutions Architect), years of experience, country, and preferred locations. This file is the sole basis for subsequent AI matching—the more specific, the more accurate the filtering.

The MCP server runs locally and exposes 10 tools, including sweep_jobs (batch scraping), get_job (read a single job), track_job (record tracking), proxy, etc. The author specifically notes that MCP is not a general-purpose large model, but a server with your personal context that knows where data comes from and what rules to process by.

What is MCP (Model Context Protocol): It is an open protocol for connecting large model clients (such as Claude Desktop, Claude Code) to external tools and data sources in a standardized way. An MCP server exposes a set of "tools" to the model, each with a name, description, and JSON Schema parameter definition. The model decides during conversation which tool to call and what parameters to pass based on user intent; the client executes and returns results to the model. Compared to hardcoded scripts, MCP's advantage is that tools are reusable and composable, and the model can autonomously plan call sequences. The 10 tools like sweep_jobs, get_job, track_job in the article are registered this way—Claude decides during scoring to first scrape, then read each one, and finally output rankings.

Having Claude Score Against Resume: From 5,200 to 5

Scraping is only the first step. The real value of AI lies in the filtering stage. The author connects to the local MCP server in Claude via the claude mcp add command, confirms successful connection, then enters a prompt: use the data_impulse tool to view the latest scraping results, find the 5 best-matching positions starting from profile.md (max 1 per company), use get_job to read each job's details, give a 0-100 match score with a one-sentence reason, and finally output a ranking table and usage receipt.

Claude's execution includes loading tool schemas, reading project context, deduplicating by company, reading and scoring each job. In the final Top 5, the first is a Senior Staff Cloud Platform Engineer at some company, scoring 92, located in Canada or US remote, no visa sponsorship needed. More importantly, the rejection logic: a position prioritizing Berlin office with relocation and visa sponsorship requirements was excluded, as was a junior/mid-level position—because profile.md explicitly states only remote and matching experience requirements.

This is exactly the author's repeatedly emphasized point: do not let AI auto-submit resumes. Bulk applications will only be directly rejected by AI screening tools in the market, which is meaningless. AI should help you pick out the few from thousands of listings that are worth applying to personally.

Why "Deduplicate by Company" and "Read Details One by One" Are Two Separate Steps: Job lists obtained during scraping typically only have summary fields like title, company, and location—insufficient for judging match quality. The detail pages read by get_job contain key information such as seniority requirements, visa sponsorship, and remote policy. If you let the model score summaries directly, it's easily misled by titles. The author limits "max 1 per company" in the prompt to prevent multiple similar positions at the same company from crowding out Top 5 slots, spreading opportunities across more employers. This two-stage process of "coarse filtering and deduplication, then detailed reading and scoring" is also a common approach for AI Agents handling large candidate sets, significantly reducing token consumption and misjudgment rates.

Cost and Practical Advice

The proxy cost for scraping over 5,200 job listings in the demo was about $0.0004 (at $1/GB), with total demo spending of $0.44. The bill shows per-site request counts, traffic, and costs—some requests even incurred no charge. For individual job seekers, this cost is almost negligible.

A few practical tips: profile.md should be as specific as possible, clearly stating remote/on-site, experience level, and location preferences so AI rejection reasons are reasonable; add or remove sources as needed, prioritizing job boards with public APIs for more stable scraping; the final job list is exported as Excel with match scores and original links for easy manual verification and application.

The value of this approach isn't "full automation," but handing the two most tedious stages—information collection and initial screening—to the Agent, while leaving decisions and applications to humans. For engineers looking for DevOps or cloud platform roles, it's a practical project worth reproducing.

One-line Summary

Developer builds AI Agent with MCP and residential proxies to scrape thousands of DevOps jobs, letting Claude score and filter against resume.

Paragraph Summary

The article introduces a job filtering system built by a developer using MCP protocol: scraping thousands of DevOps-related positions from job boards like Greenhouse, Lever, Ashby via residential proxies, then having Claude score and rank each against target roles and preferences in profile.md, ultimately selecting the 5 best matches from over 5,200 listings. Controlled experiments show that without a proxy, 362 requests were rejected as bots, while with residential proxies all 1,000 pages succeeded with zero errors. The author explicitly opposes AI auto-submitting resumes, believing AI should only handle information retrieval and initial screening, leaving decisions and applications to humans. Proxy costs are pay-per-traffic, with scraping over 5,200 jobs costing only about $0.0004 in the demo.