8,000 RMB Foreclosure Property Data Scraping Gig: A Hands-On Reverse Engineering of a JS Encrypted Parameter

Foreclosure property data scraping is a classic freelance reverse engineering scenario. Bilibili creator "Luo Ge" (洛哥) recently took on an 8,000 RMB client project. The target site had parameter encryption and risk-control mechanisms, but after debugging, the core challenge turned out to be just one thing — the sign parameter in the request header. This article fully reconstructs the analysis process for this gig, from packet capture and localization to recovering the encryption logic. It's a useful reference for developers learning web scraping and looking to get into real commercial projects.

Breaking Down the Requirements: What This Gig Actually Involves

The client's requirements were clear: scrape data from a foreclosure property website, build a GUI application (mentioned as the PLQ65 interface), scrape twice a day at 200,000 records per run, accumulating 400,000 records daily. The data covers property listings from platforms like Alibaba and JD.com. The client provides an IP pool and requires random rotation.

From a volume standpoint, 400,000 records isn't that large. The creator did the math: there are over 80,000 seconds in a day, so at 30 records per second, 200,000 records can be collected in two hours. The real problem is anti-scraping mechanisms — normal requests may hit encrypted parameters and scenario-based risk control, and batch runs will trigger CAPTCHAs.

Gigs like this command a decent price on the freelance market because they involve real risk-control countermeasures rather than simple static page parsing. The creator also notes in the video that mastering this skill set "has at least some commercial value."

Initial Packet Capture: Locating the Encrypted Parameter on the List Page

Opening the property list page and paging through with F12 packet capture revealed only one s parameter in the request, with the Data portion in plaintext — it looked unencrypted. The creator's first reaction was "doesn't seem to be any special parameter."

But copying the parameter value into the search box for a global search failed to locate it — no results. Searching with Chinese keywords also yielded nothing. The creator mentions a detail: using AI-assisted localization (including trying MCP) also took a long time without success. In the end, he went back to the traditional method — setting an XHR breakpoint in the source panel and then paging through, which finally caught the request.

After the breakpoint hit, the call stack showed that Data indeed existed. Following it further down led to a JS file. In that file's execution context, the value of the s parameter had already been generated — meaning the encryption logic lives in this file.

XHR breakpoints (XMLHttpRequest Breakpoint) are a debugging technique in browser developer tools used to pause execution when JavaScript initiates an Ajax request. Specifically, you add a breakpoint in the XHR/fetch Breakpoints area of the Sources panel, optionally specifying a keyword contained in the URL. When the page initiates a matching XHR request, the code automatically pauses at the call site that initiates the request (usually the send method). This lets you inspect the call stack and variable state at that moment before the request is sent, and trace back to the upstream code that generated the parameters. Compared to blindly searching through source code, XHR breakpoints are especially suited to scenarios where parameters are encrypted or dynamically generated and cannot be located via direct global search — one of the most commonly used methods for locating encryption entry points in scraping reverse engineering.

Breakpoint Debugging: Recovering the sign Generation Algorithm

The next key step was finding the exact location where the s parameter is computed. The creator stepped through the call stack, setting breakpoints to observe variable values.

After the breakpoint hit, checking a few key variables: the first value is a token, currently empty; s is a timestamp-like value (starting with 125), possibly a fixed parameter; data is the request body. The core logic emerged — concatenate the token, timestamp, and request data, then apply MD5 to produce the final s parameter.

The creator specifically confirmed the algorithm was standard: looking from the top of the call stack, it's a fairly conventional MD5 implementation, not a modified version. That means there's no need to reverse-engineer a custom hash logic — standard MD5 reproduces it.

MD5 (Message-Digest Algorithm 5) is a widely used hash function that outputs a 128-bit (16-byte) digest, typically represented as a 32-character hexadecimal string. It exhibits the avalanche effect: changing even a single character of the input dramatically changes the output, so the original text cannot be inferred from the hash value. In web signing scenarios, a common approach is to concatenate multiple parameters in an agreed order and compute MD5; the server recomputes it the same way and compares, verifying the request hasn't been tampered with. Note that MD5 is no longer considered a secure cryptographic algorithm (collision attacks exist), but it's still widely used for tamper-proof signatures and API authentication. When reproducing it, the key isn't the algorithm itself but the original concatenated string: delimiters, parameter order, casing, timestamp digit count, and other details must exactly match the server side.

The call stack is the mechanism the JavaScript engine uses to track function call relationships: each time a function is called, its execution context is pushed onto the top of the stack, and popped when the function returns. When paused in the DevTools Sources panel, the Call Stack area lists the entire chain in the order "current function → caller → caller's caller." During reverse engineering, you can click up through stack frames from the breakpoint hit location, inspecting local variables and parameters at each level to determine which layer and which inputs produced the encrypted parameter. This bottom-up backtracking approach lets you progressively reconstruct the full parameter generation logic even when no source is readable and variable names are obfuscated.

Code Reproduction: Pitfalls with Concatenation and MD5

Moving to implementation, the creator demonstrated parameter construction live: token generated from the current timestamp, s set to a fixed value, data as the request body, and the three concatenated and hashed with MD5.

The first test didn't match — the output started with 30916 while the target value started with 7A. Investigation revealed two problems: one, a symbol's format was wrong in the concatenation (the target value contained fragments like 7A and 677), and two, the timestamp was written incorrectly. After fixing the parameter format, the MD5 result matched the target.

Here's a lesson worth recording: MD5 itself is a standard algorithm, but if the delimiter, parameter order, or timestamp precision in the concatenated string is off by even a bit, the result is completely different. When debugging, always compare intermediate values segment by segment rather than only looking at the final hash.

In Python, reproducing an MD5 signature typically uses the hashlib library: hashlib.md5(concatenated_string.encode('utf-8')).hexdigest() yields a 32-character lowercase hexadecimal digest; if the target value is uppercase, call .upper() afterward. The most common pitfalls during reproduction include: encoding mismatch (UTF-8 and GBK produce different byte sequences for Chinese request bodies), mismatched concatenation order and delimiters, wrong timestamp units (seconds vs. milliseconds differ by a factor of 1000), and hexadecimal case differences. It's recommended during debugging to print out the original string to be hashed and compare it character by character against the concatenated result seen in the browser breakpoint, confirming an exact match before computing the hash — this quickly pinpoints the source of any discrepancy.

Detail Page: Static Page, Parse Directly

Having solved the s parameter on the list page, the detail page turned out to be simpler. Refreshing the detail page API revealed it was static with no additional encrypted parameters — a direct HTML request and parse was enough to get the data.

The technical path for the whole project became clear: the list page requests paginated data with an MD5 signature, and the detail page directly parses static HTML. The core effort centers on recovering the signature parameter; the rest is routine scraping framework setup and IP rotation logic.

Takeaways from This Case

From a technical standpoint, this gig's encryption strength isn't high — a single MD5 signature with no obfuscation, no environment detection, no slider CAPTCHA. But the creator also notes in the video that batch scraping would still encounter CAPTCHAs and other risk controls, and actual delivery requires handling these edge cases.

For learners, the value of this case lies in fully demonstrating the analysis workflow of a freelance project: packet capture to observe parameters → failed search localization → XHR breakpoint backtracking → call stack variable analysis → code reproduction and verification. The creator also admits that AI tools "may not be able to figure it out for you" in this kind of scenario, and that traditional debugging techniques remain fundamental skills.

It should be noted that this article only documents the technical analysis process. Actual scraping development should comply with the target website's robots.txt and relevant laws and regulations, and data collection requires authorization.

One-line Summary

Full reverse engineering of the sign parameter and code reproduction for an 8,000 RMB foreclosure property scraping gig.

Paragraph Summary

This article fully reconstructs the technical analysis of an 8,000 RMB foreclosure property data scraping project. The client needed 400,000 property records daily from platforms like Alibaba and JD.com, with an IP pool for random rotation. The core challenge was the sign parameter in the list page request header: after global search failed to locate it, the author used XHR breakpoints and call stack backtracking to find the encryption logic in a JS file. Breakpoint analysis confirmed sign is generated by concatenating token, timestamp, and request body with standard MD5 — not a modified algorithm. During code reproduction, mismatched concatenation symbol format and timestamp errors caused results to differ; after fixes, the result matched the target. The detail page is static HTML and can be parsed directly. The author also notes that AI-assisted localization was limited, traditional breakpoint debugging remains a fundamental skill, actual delivery still requires handling CAPTCHAs and other risk-control edge cases, and emphasizes compliance with robots.txt and relevant regulations.