OSINT Tools for Automated Discovery of Exposed Files on Domains: Principles, Applications, and Compliance Boundaries

Data Breaches Often Start with a Forgotten File

In the cybersecurity world, many serious data breaches don't originate from sophisticated hacking attacks but from files inadvertently exposed in publicly accessible locations—a database backup that was never deleted, a configuration file with hardcoded keys, or an internal document mistakenly uploaded to a public directory.

Recently, an OSINT (Open-Source Intelligence) tool appeared on Hacker News, specifically designed to automatically discover sensitive files exposed on domains. OSINT was originally an intelligence concept that originated during the Cold War era for military intelligence gathering—first systematically employed by the CIA and British intelligence agencies, who analyzed publicly available Soviet academic journals, government bulletins, and broadcast content to infer military and technological developments. The 1992 amendment to the U.S. National Security Act formally classified it as an independent intelligence category alongside SIGINT (Signals Intelligence) and HUMINT (Human Intelligence). In the digital age, OSINT's scope has expanded dramatically—the vast amount of publicly available data on the internet has made it a core reconnaissance method in cybersecurity. From the early Shodan network device search engine (launched in 2009), to Maltego relationship graph analysis, the Recon-ng framework, and today's AI-powered automated intelligence aggregation, a complete passive reconnaissance ecosystem has formed spanning domain enumeration, certificate transparency log analysis, social media intelligence, dark web monitoring, and more. Notably, the "open source" in OSINT doesn't refer to open-source software code, but rather "open source" in the intelligence sense—meaning any legally and publicly accessible information channel. This semantic distinction is frequently confused outside security circles.

While this tool hasn't yet gone viral, the problem it addresses is a pain point that every security team cannot avoid: Do you truly know which files on your domain are publicly accessible?

This article will break down the tool's technical logic, practical application value, and the compliance and ethical boundaries that must be observed when using such tools.

What Problem Does This OSINT Tool Solve?

What Are "Exposed Files on a Domain"?

"Exposed Files" refers to files that should not be publicly accessible but have been placed on network-accessible paths due to configuration oversights or human error. Common high-risk types include:

• Backup files: Such as .sql, .bak, backup.zip, which may contain complete database contents.
• Configuration files: Such as .env, config.php, .git/config, which may leak database credentials and API keys.
• Log files: Which may record user behavior, error stacks, or even sensitive parameters.
• Version control directories: An exposed .git directory can be used to reconstruct complete source code. .git directory exposure is a particularly severe risk—Git's object storage mechanism preserves the complete history of all commits in a project. Even if a developer realizes a commit contained hardcoded keys and subsequently deletes them, those keys still exist in the Git history and can be fully recovered using git log and git show commands. Attackers can use git clone or specialized tools (such as GitTools, git-dumper) to completely reconstruct project source code from an exposed .git directory—a fully automated process taking no more than a few minutes. In 2023, a report from security research firm GitGuardian showed that over 10 million new hardcoded secrets are added to public code repositories each year, and exposed .git directories mean that similar issues in private repositories also face the risk of passive disclosure. The damage far exceeds that of a single file leak.
• Temporary and debug files: phpinfo pages, test scripts, and other artifacts left over from development.

Once these files are indexed by search engines or detected by automated scripts, they can become an attacker's entry point.

The Core Value of Automated Discovery

Traditionally, security researchers needed to manually construct URL paths and probe them one by one to discover such files. The value of this OSINT tool lies in automating the entire process: input a target domain, and the tool batch-probes based on a dictionary of common sensitive paths, filtering out files that actually exist and are accessible. This dramatically lowers the barrier to asset inventory, enabling defenders to proactively audit their own exposure surface from an attacker's perspective.

Technical Principles Explained

Dictionary-Driven Path Probing

The core mechanism of such tools follows a three-step approach: "dictionary + request + response evaluation." The tool contains a built-in dictionary of thousands of common sensitive filenames and paths, issues HTTP requests for each entry against the target domain, then comprehensively determines whether a file is truly exposed based on signals such as status codes (200 indicating the resource exists), response content length, and Content-Type.

Dictionary quality directly determines scanning effectiveness. The most widely used dictionaries in the industry come from the SecLists project—an open-source dictionary collection initiated and maintained by security researcher Daniel Miessler in 2012. It has become one of the highest-starred security tool repositories on GitHub (over 50,000 stars), covering dozens of security testing scenarios including directory brute-forcing, subdomain enumeration, password cracking, and fuzzing payloads. It is one of the most frequently referenced foundational resources in penetration testing. Its content derives from accumulated real penetration testing data, CVE vulnerability analysis, and continuous community contributions from security researchers worldwide. Among specialized file discovery dictionaries, raft-large-files.txt contains approximately 37,000 paths verified through real penetration tests with the highest coverage rate, while common.txt is known for its streamlined efficiency, suitable for quick initial screening. Notably, even with the same scanning engine, discovery rates can vary by several times depending on the dictionary set used—this is why top bug bounty hunters often invest significant time maintaining their own private dictionary libraries. Modern tools also support intelligent dictionary switching based on the target's technology stack—by identifying Server, X-Powered-By, and other fields in response headers to determine whether the target uses WordPress, Laravel, or Django, then loading the corresponding specialized dictionary to significantly improve hit rates.

Key Mechanisms for Reducing False Positives

Relying solely on HTTP 200 status codes is insufficient, because many websites configure "soft 404s"—returning 200 with a custom error page for non-existent pages. "Soft 404" is one of the trickiest sources of false positives in web security scanning, particularly prevalent since the adoption of modern single-page application (SPA) frameworks like React, Vue, and Angular—these frameworks typically handle routing on the client side, with the server uniformly returning index.html with a 200 status code for any path, leaving it to frontend JavaScript to determine what to render. This causes traditional status-code-based scanning logic to completely fail. Mature detection tools typically employ multiple validation layers: before the formal scan, they request several random paths that certainly don't exist, recording their response length and content hash as a "baseline response fingerprint." During scanning, responses highly similar to the baseline are automatically filtered out. Additional mechanisms include:

• Comparing response content similarity against known error pages
• Verifying that the returned file's MIME type matches expectations
• Checking file magic bytes to confirm file authenticity

The concept of file magic bytes originated from the Unix /etc/magic database, created by Ian Darwin in 1986 for the file command. Magic bytes are specific byte sequences at the beginning of a file that identify its true format: ZIP files start with 50 4B 03 04, PDFs with 25 50 44 46, SQLite databases with 53 51 4C 69, and Windows executables (PE format) with 4D 5A (the characters MZ). This mechanism doesn't rely on file extensions, so even if an attacker renames a file, tools can still identify its true type. In forensic analysis and malware detection, antivirus engines similarly prioritize magic bytes for format identification to circumvent rename-based evasion. By verifying these bytes, tools can eliminate false positives where a "200 response is actually an HTML error page," ensuring every finding in the report is a genuine file. Over 4,000 known file magic signatures exist, with Gary Kessler's File Signatures Table being the industry's most authoritative reference database. The false positive rate directly determines a tool's practical value—a scan result with too much noise actually drowns out the risk signals that truly need attention.

Combining with Google Dorking

A more advanced OSINT approach combines active probing with search engine intelligence: using Google advanced search operators like filetype: and site: to locate already-indexed exposed files, then supplementing with batch probing to achieve dual coverage of passive intelligence and active scanning, significantly improving discovery efficiency.

Google Dorking is the technique of using search engine advanced operators to locate public information. It was first systematically organized by security researcher Johnny Long in 2001 and published as the Google Hacking Database (GHDB)—this database contains over 6,800 categorized Dork queries and is currently maintained by the Offensive Security community, integrated into the Exploit-DB platform. GHDB's unique value lies in representing a "zero-noise" passive intelligence model: all discoveries come from content that search engines have already crawled and indexed, meaning the target server receives no direct requests from the researcher. Common operators include: site: to restrict to a domain, filetype:/ext: to filter file types, inurl: to match path keywords, and intitle: to match page titles. It's worth noting that automated bulk Dorking faces two layers of constraints: search engine ToS typically prohibit automated scraping, and Google's rate-limiting mechanisms make large-scale automation difficult. Some researchers turn to professional intelligence platforms like Shodan and Censys APIs as alternatives. When combined with active probing, strict adherence to authorization boundaries is required.

Application Scenarios and Practical Value

Attack Surface Management (ASM)

For enterprise security teams, such tools are an indispensable component of the Attack Surface Management workflow. Attack Surface Management emerged as an independent security category rapidly between 2019-2021, with Gartner listing it as an important emerging direction in cybersecurity. ASM's core proposition is that enterprise security teams often lack a complete view of their own digital assets. According to CyCognito's 2023 industry report, the average mid-sized enterprise has over 500 internet-exposed assets, with approximately 30% outside IT department oversight—shadow IT, legacy domains, and third-party SaaS integration points continuously expand the actual exposure surface. "Shadow IT" refers to systems and services built without formal IT department approval, set up by business units or individuals—for example, landing pages self-built by marketing teams or cloud storage buckets registered under individual developer accounts. These assets often exist outside security governance and represent ASM's hardest-to-cover blind spots. Leading ASM vendors (Censys focuses on certificate and protocol-layer scanning, Runzero specializes in internal network asset discovery, CyCognito focuses on attacker-perspective simulation) have all secured hundred-million-dollar funding rounds, reflecting capital markets' strong endorsement of this direction. This open-source tool represents a lightweight implementation of this concept, enabling small and mid-sized teams to gain similar capabilities at minimal cost. Regular scanning of owned domains allows discovering forgotten exposed files and completing remediation before attackers do.

Bug Bounties and Penetration Testing

For bug bounty hunters and penetration testers, exposed files are often the "gold mine" of the information gathering phase. A leaked .env configuration file might directly provide database credentials for system entry, converting a reconnaissance-phase discovery directly into a demonstrable high-severity vulnerability. In historical reports on major bug bounty platforms like HackerOne and Bugcrowd, exposed configuration files and .git directories are among the most frequent vulnerability types to receive high bounty payouts, with some cases exceeding ten thousand dollars, confirming the real risk value of such discoveries.

Security Awareness Education

The mere existence of such tools serves as a warning: any file placed in a web root directory should be considered "potentially public." It reminds developers and operations staff that sensitive data storage must strictly follow the principle of minimum exposure, rather than relying on the wishful thinking that "nobody knows this path"—this mindset is known in security as "Security through Obscurity," a widely recognized anti-pattern, because the existence of automated scanning tools fundamentally invalidates the assumption that "path obscurity" provides protection.

Compliance and Ethical Boundaries Must Not Be Ignored

It must be particularly emphasized: OSINT tools are a double-edged sword. Conducting large-scale probing against someone else's domain without authorization may cross legal red lines. Although these tools essentially just send HTTP requests—technically no different from normal web browsing—when the behavior exhibits systematic, batch-scanning characteristics, it may be classified as a precursor to unauthorized access. In the United States, the Computer Fraud and Abuse Act (CFAA) has historically been controversial in defining unauthorized access, with some precedents classifying bulk automated requests as "access exceeding authorized scope" even when the target server is publicly accessible. In China, Article 27 of the Cybersecurity Law similarly explicitly prohibits unauthorized network probing activities. Practitioners must thoroughly understand the legal boundaries of their jurisdiction.

Responsible use should follow these principles:

• Only scan assets you own or have received explicit written authorization for
• In bug bounty programs, strictly adhere to the defined testing scope
• Control request frequency to avoid causing DoS-like pressure on target servers
• When discovering third-party vulnerabilities, follow the Responsible Disclosure process

Responsible vulnerability disclosure is an industry norm formed by the cybersecurity community, with its philosophy gradually refined by security experts such as Bruce Schneier in the 2000s, evolving from "full disclosure" to "coordinated disclosure." The 90-day disclosure window was formally established and publicly enforced by Google's Project Zero team in 2014. Its background was the security community's longstanding dissatisfaction with inconsistent "coordinated disclosure" practices—some vendors would indefinitely delay patch releases under the guise of "still fixing," leaving vulnerabilities in a known-but-unpatched dangerous state for extended periods. Project Zero's hardline stance (mandatory public disclosure upon deadline expiration, regardless of whether the vendor has completed a fix) sparked widespread industry debate but was ultimately proven effective in accelerating overall vulnerability fix speeds. In 2019, major vendors including Microsoft, Google, and Apple jointly published a "Coordinated Vulnerability Disclosure Framework" establishing 90 days as the industry standard, along with a 7-day emergency disclosure clause for critical zero-day vulnerabilities. The underlying logic is that 90 days gives organizations willing to fix issues sufficient time to develop and deploy patches, while preventing indefinite shelving. Today, many organizations have established formal VDPs (Vulnerability Disclosure Policies) through platforms like HackerOne and Bugcrowd, specifying submission channels, response timelines, and reward scopes. This makes the process more standardized and provides good-faith researchers with a degree of legal protection—provided the researcher doesn't exceed the boundary of "discovery" (i.e., doesn't attempt to exploit the vulnerability to obtain data).

A healthy ecosystem for open-source security tools is built upon the self-discipline of every user.

Proactive Defense: Examining Yourself from an Attacker's Perspective

This OSINT tool that appeared on Hacker News represents a security philosophy worth promoting—using automated means to examine your own exposure surface from an attacker's perspective, which is the core of modern proactive defense thinking. This philosophy is commonly summarized in the industry as the "Assume Breach" principle: rather than operating on the premise of "whether we'll be attacked," it presumes that attackers have already or could at any moment gain entry, thereby forcing defenders to continuously examine their own exposure surface and response capabilities. For security practitioners, rather than waiting for exposed files to be maliciously exploited, it's better to take the initiative and proactively audit.

The tool itself isn't complex. Its true value lies in continuously reinforcing a security truism that has been repeatedly validated: The simplest oversights often cause the most severe consequences. Regular, systematic inspection of your own assets is the most effective way to prevent such oversights from escalating into data breach disasters.