Reverse-Engineering ChatGPT's Source Selection: What Network Traffic Analysis Reveals About AI Citations

Why Study Network Traffic Instead of Observing the Output

When we ask ChatGPT a question that requires real-time information, it activates its web browsing capability, pulling content from the internet and synthesizing it into an answer. But one key question continues to puzzle researchers and content creators alike: How exactly does ChatGPT decide which sources to cite?

Most analyses stop at observing ChatGPT's final output—namely, the citation links listed at the end of a response. This approach has clear limitations: the output has already been filtered, ranked, and rephrased by the model, and what's presented to the user is just the "tip of the iceberg." It fails to accurately reflect what actually happens at the retrieval layer.

A more rigorous path is to directly monitor and parse the network traffic ChatGPT generates during web retrieval. Through packet capture analysis of the actual HTTP requests it sends, the domains it accesses, and the data structures it returns, we can reconstruct the raw information the model truly encountered during its "thinking" process—rather than the processed version shown to users.

The packet capture analysis referred to here is a classic technique in the fields of network security and performance debugging. It originated from network debugging practices on ARPANET in the 1970s, first becoming popular in Unix systems through command-line tools like tcpdump, and later becoming a standard tool for security research as the internet commercialized. In 1998, Gerald Combs developed Ethereal (later Wireshark), bringing graphical packet capture capabilities to a broader community of engineers, expanding the technique from the exclusive domain of network administrators into a general-purpose tool for penetration testing, protocol analysis, and application debugging. Modern packet capture toolchains fall into two categories: passive sniffing types (such as Wireshark, which listens to all packets passing through a network interface) and active proxy types (such as mitmproxy and Charles Proxy, which participate in connection establishment as a man-in-the-middle). By intercepting packets at the network interface layer, one can reconstruct the complete communication content of the application layer.

The core principle of a Man-in-the-Middle Proxy is to insert itself between the client and server, establishing separate TLS connections with each end, thereby enabling observation of plaintext content outside the encrypted channel. For HTTPS encrypted traffic, a man-in-the-middle proxy must solve the TLS certificate trust chain problem—the security of modern TLS (Transport Layer Security) protocols is built upon the X.509 certificate system and Public Key Infrastructure (PKI). Browsers and operating systems come with hundreds of trusted root CAs (Certificate Authorities) built in, and any certificate issued by these CAs is silently accepted by the client. Researchers inject a custom root CA certificate into the system certificate store, causing forged certificates issued by the proxy to be trusted by the client, thereby achieving transparent decryption of TLS-encrypted traffic. This is the core prerequisite for observing SSL/TLS traffic.

It's worth noting that some applications implement SSL Pinning, which hardcodes the public key fingerprint or certificate hash of the server certificate in the client code, refusing to accept certificates issued by unexpected CAs—even if that CA is already in the system trust store. This mechanism is widely used in financial apps and high-security mobile applications for anti-man-in-the-middle protection. Researchers typically need to use dynamic instrumentation tools like Frida to hook system functions related to SSL verification at runtime (such as iOS's SecTrustEvaluate or Android's X509TrustManager), replacing the certificate verification logic with a stub function that always returns "trusted," thereby bypassing this protection. For ChatGPT's web traffic, since browser trust systems are relatively open, SSL Pinning usually doesn't pose a major obstacle—this is a key prerequisite for analyzing the network behavior of AI products like ChatGPT. This method already has mature practices in mobile app reverse engineering, ad SDK auditing, and other fields, and in recent years has been introduced into the field of AI behavior analysis, becoming an important tool for studying the "external behavior" of large models.

What Network Traffic Analysis Reveals

The Separation of the Retrieval Layer and the Generation Layer

Through traffic analysis, we can clearly see that ChatGPT's web retrieval is divided into two stages.

The first stage is the recall of candidate sources: the model generates retrieval queries based on the user's question, sends requests to the search backend, and obtains a batch of candidate web pages or data sources. The second stage is the fetching and filtering of content: selecting some of the links from the candidate set for actual fetching, and then having the language model synthesize the content.

The academic name for this architecture is Retrieval-Augmented Generation (RAG). RAG was formally proposed by Meta AI in the 2020 paper "Retrieval-Augmented Generation for Knowledge-Intensive NLP Tasks." Its core idea is to decouple external knowledge retrieval from language model generation: the model no longer relies solely on the parametric knowledge solidified during training, but instead dynamically queries external data sources during inference, injecting the retrieval results as context into the generation process. This design directly addresses two fundamental limitations of large language models—the training cutoff and hallucination: by injecting real external documents during inference, the model can generate answers with traceable evidence, rather than relying purely on the statistical patterns compressed and stored in its parameters.

RAG has undergone significant engineering evolution from an academic concept to industrial deployment. Early implementations relied on Sparse Retrieval, whose representative algorithm BM25 (Best Match 25) is an improved version of TF-IDF. It computes keyword matching scores through term frequency saturation functions and document length normalization, performing robustly in exact keyword matching scenarios, but with limited recall for semantic variants such as synonyms and hyponyms/hypernyms. Modern production-grade RAG commonly adopts Hybrid Retrieval, weighting and fusing the scores of BM25 sparse retrieval with Dense Retrieval—the former ensures high recall for exact matches, while the latter addresses the vocabulary gap problem through semantic similarity. Dense retrieval uses an Embedding Model (such as OpenAI's text-embedding-3 series, Google's Gecko, or the open-source BGE and E5 series) to encode text into high-dimensional semantic vectors, achieving millisecond-level semantic matching across document repositories of tens of millions through Approximate Nearest Neighbor search (ANN, such as HNSW graph algorithms or IVF-PQ quantization indexing). The Reranking layer typically uses a Cross-Encoder model (such as Cohere Rerank, BGE-Reranker) to finely score the recalled candidates. The essential difference between a Cross-Encoder and a Bi-Encoder (i.e., the embedding model) is that the former concatenates the query and document before jointly encoding them, capturing fine-grained interaction features, but at higher computational cost. Therefore, it is usually only used for the secondary fine-ranking of the Top-K candidates, rather than a full-repository scan. In terms of engineering implementation, vector databases such as Pinecone, Weaviate, and Qdrant have become mainstream choices for building production-grade RAG systems, each making its own trade-offs across engineering dimensions such as distributed storage, real-time updates, and hybrid filtered queries.

In OpenAI's implementation, this architecture typically includes four stages: Query Rewriting, vector or keyword retrieval, content extraction and Reranking, and final generation. The query rewriting stage is particularly critical—the model transforms the user's natural language question into a normalized query form more suitable for retrieval. For example, it rewrites "what's going on with Tesla lately" into a query expression like "Tesla 2024 financial results latest news" that is better suited for search engine indexing. The network requests produced by this step are often clearly identifiable at the traffic layer. What network traffic can observe is precisely the two stages that produce outbound HTTP requests—"retrieval" and "fetching"—while the generation stage occurs internally on the server and cannot be directly observed at the network layer.

This means that the sources ultimately cited are just a subset of the recall set. A large number of sources that were retrieved but not cited are clearly visible at the traffic layer, yet quietly "disappear" in the output. The real filtering happens where the user cannot see it—this is crucial to understanding ChatGPT's source selection preferences.

Domain Preferences and Traffic Distribution Patterns

Traffic data can also expose the model's preference patterns for specific sources: Which domains are accessed frequently? Which types of websites more easily enter the candidate set? This information cannot be fully inferred from the output citation list, because the final output often retains only a few "representative" links.

Based on existing observation cases, government websites (.gov), academic institutions (.edu), mainstream news media, and knowledge aggregation platforms like Wikipedia tend to occupy a higher proportion of the candidate set, showing a certain structural similarity to traditional search engines' preference for authority. However, unlike traditional search, the AI retrieval layer is more inclined to access pages with high content density and clear structure, rather than relying solely on the number of external links as an authority signal. This difference can be understood from the intrinsic incentive structure of RAG systems: the language model needs to inject the highest information-density document fragments within a limited Context Window, which naturally favors structured, clearly-paragraphed content with a moderate density of terminology, while pages that rely on visual layout, JavaScript dynamic rendering, or fragmented information are at a natural disadvantage in machine readability.

It's worth noting that the capacity limit of the context window (measured in Token count, with mainstream models evolving from the early GPT-3's 4K Tokens to today's million-Token scale of Claude and Gemini) directly determines how much external document content a RAG system can inject. In the era of limited context windows, retrieval precision and document extraction strategies (Chunking Strategy) were crucial—how to slice a long article into fragments that both maintain semantic integrity and are suitable for embedding retrieval is one of the core challenges in RAG engineering practice. As long-context models become widespread, this constraint is gradually being relaxed, but retrieval precision still has a significant impact on final generation quality. The "Lost in the Middle" effect (the phenomenon where models pay less attention to content in the middle of the injected context than at the beginning and end) reminds us that the ranking position of content in the retrieval results still affects the probability of it being effectively utilized.

For practitioners concerned with SEO and content distribution, this type of analysis has direct practical value: If you want your content to be more likely cited by AI, understanding the recall logic of the retrieval layer is far more instructive than staring at the final citation list and reasoning backward.

The Value of This Research Methodology

A Feasible Path to Breaking Open the "Black Box"

The web browsing capabilities of large language models are often viewed as an uninterpretable black box. But network traffic analysis offers a pragmatic way to break the deadlock: Even without access to the model's internal weights, you can still infer its operating mechanisms by observing its interaction behavior with the external world.

AI Explainability / Interpretability is one of the core research topics in academia today, with mainstream approaches falling into two directions: "intrinsic interpretability" and "behavioral interpretability." Representative work in intrinsic interpretability methods includes Anthropic's research on Mechanistic Interpretability—this research direction attempts to locate the storage positions of specific knowledge or reasoning behaviors at the level of neural network activations, decomposing the activation space of MLP layers through Sparse Autoencoders to identify Feature Directions corresponding to specific concepts. Its representative achievement "Towards Monosemanticity" identified millions of interpretable features in the Claude model. Sparse autoencoders are effective in this scenario because the internal representations of neural networks often exhibit "Polysemanticity"—a single neuron may respond to multiple semantically unrelated concepts simultaneously, and sparse autoencoders, by enforcing activation sparsity, can decompose these superimposed representations into feature vectors closer to being monosemantic, allowing researchers to more precisely trace the flow paths of specific concepts within the model. However, such intrinsic interpretability methods face a fundamental dilemma: for commercial closed-source models like GPT-4 and Gemini, researchers cannot access the model weights and activation values, and the intrinsic interpretability path is almost entirely closed off.

Behavioral Interpretability methods, on the other hand, focus more on the systematic output patterns of the model under specific inputs. Core tools include Adversarial Prompting, systematic Ablation Studies, and Black-box Probing, none of which rely on direct access to parameters. Network traffic analysis belongs to a special form of the behavioral interpretability path—what it observes is not language-level input-output pairs, but the network behavior generated when the model interacts with the external world, sharing the same lineage as Dynamic Analysis and Fuzzing in the software security field. The credibility of this method comes especially from its reproducibility: any researcher with the corresponding technical capabilities can independently verify the observation results, forming a consensus understanding of AI system behavior, thereby compensating for the fundamental limitation that intrinsic interpretability methods have almost no way to approach closed-source models.

Network traffic is the "interface layer" between the model and the internet, and this layer is relatively transparent and measurable. This "behavioral observation" approach shares a common thread with black-box testing and reverse engineering in security research—rather than relying on technical documentation published by vendors, it builds an understanding of the system through empirical data, offering high credibility and strong reproducibility.

Practical Implications for Content Creators

As more and more users obtain information through AI, "being cited by AI" is becoming an emerging traffic entry point. Traditional SEO focuses on how to achieve high rankings on search engine results pages, while content optimization for AI requires understanding the underlying logic of the model's retrieval and source selection.

This emerging field is academically termed Generative Engine Optimization (GEO), formally proposed around 2023 by researchers from institutions such as Princeton University and the University of Chicago. The related paper "GEO: Generative Engine Optimization" attracted widespread attention in academia.

GEO and traditional SEO have fundamental differences in optimization objectives and signal systems. The core algorithm of traditional SEO—PageRank and its successors (such as TrustRank and the HITS algorithm)—is essentially an authority propagation model on a directed link graph, whose mathematical basis is Random Walk and the stationary distribution of Markov chains: a page's authority score equals the sum of the weighted contributions from all pages pointing to it, weighted by their own authority scores. The core assumption of this model is that "a page cited by authoritative pages is itself authoritative," so the quantity and quality of external Backlinks is the strongest ranking signal, and Link Building around PageRank signals evolved into the core practice of traditional SEO. GEO, on the other hand, faces an "audience" that is a language model—a probability distribution system pre-trained on massive text with the Next-Token Prediction objective, whose "trust" mechanism is essentially closer to a projection of the statistical characteristics of training data: content patterns that frequently co-occur with authority markers (such as "research shows," "according to data from XX institution") in the pre-training corpus will make the model more inclined during inference to judge similarly-styled content as citable sources. Therefore, keyword density stuffing and low-quality external links piled up for SEO not only have no positive value in the GEO context, but may even backfire by triggering the model's recognition of low-quality content patterns.

Preliminary research shows that citing authoritative sources, using clear structured statements, adding statistical data and specific facts, and improving the machine readability of content (such as schema.org structured data markup) are effective GEO strategies. Among these, schema.org markup deserves special mention: this is a semantic markup specification jointly promoted by mainstream search engines such as Google, Microsoft, and Yahoo. By embedding structured descriptions in JSON-LD or Microdata format within HTML, crawlers can unambiguously identify key entities in a page (such as author, publication date, article category, rating data, etc.). This machine-readability friendliness also has positive value for the content parsing of AI retrieval systems. JSON-LD (JavaScript Object Notation for Linked Data), as a W3C recommended standard, embeds structured semantic descriptions within the page's <script> tag, conveying rich entity relationship information to machines without modifying the main HTML structure. Compared to traditional Microdata, it has lower implementation costs and higher maintenance flexibility, and has become the preferred format for semantic markup on modern websites. In addition, timeliness annotations of content (clear publication and update dates), author authority signals (author bio, institutional affiliation, ORCID and other academic identity identifiers), and the uniqueness of content (providing original data or analysis not covered by other sources) are also considered important dimensions of GEO optimization.

This research offers a clear reminder: don't perform reverse optimization based solely on the citation list AI ultimately displays, because that is already the result after multiple layers of filtering. What truly deserves in-depth study are the rules of the retrieval recall stage—Is your content easily hit by retrieval queries? Is it located in the domain ecosystem the model prefers? Is your structured data easy to crawl?

Limitations and Extended Thinking

It must be objectively pointed out that methods like traffic analysis also have natural boundaries. First, OpenAI's retrieval backend may adjust at any time, so observed behaviors are time-sensitive. Second, network traffic can only reflect "what was accessed," and cannot directly reveal the model's internal reasoning logic during the generation stage. Third, technical measures such as encrypted traffic and server-side rendering may make some interactions difficult to fully reconstruct. Fourth, the sample size of a single observation is limited, requiring large-scale, systematic repeated experiments to draw statistically significant conclusions. In addition, OpenAI may introduce stronger Traffic Obfuscation or Request Batching mechanisms in the future—for example, merging multiple retrieval requests into a single gRPC streaming call, or proxying domain access on the server side—further reducing the signal-to-noise ratio of such analysis. gRPC is a high-performance remote procedure call framework developed by Google, based on the HTTP/2 protocol and Protocol Buffers serialization format. Its bidirectional streaming capability allows multiple logical requests to be encapsulated within a single persistent connection, appearing from the external traffic layer only as an encrypted byte stream on a single long connection, which would greatly increase the difficulty of reverse-engineering request semantics based on traffic analysis. This requires researchers to continuously update their observation methodology and combine network traffic analysis with other behavioral analysis methods (such as prompt injection probing, systematic output format testing, etc.) to build a more complete chain of evidence.

Nevertheless, starting from network traffic rather than the output result remains one of the empirical paths closest to the truth for understanding AI source selection mechanisms today. It represents a pragmatic research attitude—rather than speculating about what the model "wants to say," it's better to observe what it "actually does."

For developers, researchers, and content strategists hoping to gain a deeper understanding of the information retrieval mechanisms of generative AI, this underlying perspective provides insights far more valuable than surface-level observation.