DOGE's Use of ChatGPT to Review Grants Ruled Unconstitutional: The Legal Red Line for AI in Government Decision-Making

DOGE's Use of ChatGPT to Review Grants: What Happened

U.S. Federal Judge Colleen McMahon issued a 143-page ruling on Thursday, finding that the Department of Government Efficiency (DOGE) acted unconstitutionally when it cancelled over $100 million in federal grants. At the heart of the controversy: DOGE used ChatGPT to determine whether a grant was related to Diversity, Equity, and Inclusion (DEI), and based cancellation decisions on those results.

DOGE — the Department of Government Efficiency — is a government reform body established by the Trump administration in early 2025 under the leadership of Elon Musk. Its core mission is to dramatically cut federal spending and eliminate programs deemed inefficient or redundant. After its creation, DOGE quickly took a series of aggressive actions, including freezing federal grants, laying off government employees, and shutting down certain federal agencies. However, the department's legal standing has itself been contested — critics argue it was never formally established through Congressional authorization, making the scope of its authority a focal point of ongoing litigation.

DEI (Diversity, Equity, and Inclusion) is the other key backdrop to this controversy. After taking office in January 2025, the Trump administration signed multiple executive orders mandating the comprehensive elimination of DEI-related programs across the federal government, characterizing them as "reverse discrimination" against certain groups. This policy direction put a wide range of federal grants under review and at risk of cancellation — spanning areas from minority community services and academic research diversity to public health equity, affecting everything from university research funding to community health programs.

This case not only exposed the risks of AI tools being misused in government decision-making but also sparked a profound discussion about the boundaries of artificial intelligence applications in public administration.

How DOGE's AI Review Process Worked

Using ChatGPT as a Grant Reviewer

According to court documents, DOGE employed a jaw-dropping workflow when reviewing federal grants: they fed grant-related information into ChatGPT and had this general-purpose chatbot determine whether a project involved DEI content. Once ChatGPT returned an affirmative answer, the grant faced cancellation.

To understand why this approach is so absurd, it helps to understand the technical nature behind ChatGPT. ChatGPT is built on the GPT series of Large Language Models (LLMs), which work by statistically learning from massive text datasets to predict the next most probable sequence of words. This means it is fundamentally a probabilistic generation system, not a logical reasoning engine. The model's "hallucination" problem refers to its tendency to generate factually incorrect information with a tone of high confidence — an inherent flaw in all current large language models. More critically, LLM outputs are non-deterministic: even with identical input prompts, the model may produce different or even contradictory answers at different times due to settings like the temperature parameter. This uncertainty might be acceptable in casual conversation, but in policy review scenarios that demand consistency and reproducibility, it constitutes a fundamental reliability defect.

This approach had multiple problems:

• ChatGPT is not a policy review tool: It's a general-purpose large language model not designed for legal compliance analysis. Its outputs are stochastic — the same question may yield completely different answers at different times.
• The decision process lacked professional review: Entrusting decisions involving millions of dollars in public funds to an AI chat tool completely bypassed due process and professional review mechanisms.
• No traceability or accountability: ChatGPT's reasoning process is opaque, and unlike human decision-makers, no legal entity can be held responsible for its outputs.

Why the Judge Found This Approach "Both Foolish and Illegal"

Judge McMahon explicitly stated in her ruling that DOGE's use of ChatGPT to review grants was indefensible on multiple levels:

• Absence of procedural justice: Cancelling federal grants requires strict administrative procedures, including notifying affected parties and providing opportunities to appeal. DOGE skipped all of these legally mandated steps entirely.
• Unreliable decision basis: ChatGPT's judgments carry no legal weight and lack the capability to accurately classify policies. Using it to make consequential fiscal decisions essentially outsources government responsibility to an uncontrollable tool.
• Exceeding authority: Over $100 million in grants were cancelled, affecting numerous previously approved projects and institutions. This kind of large-scale unilateral action exceeded DOGE's statutory authority.

It's worth understanding that U.S. federal grant management is governed by a series of strict legal frameworks, primarily the Administrative Procedure Act (APA) and the Impoundment Control Act. Under these laws, once federal funds are appropriated by Congress, the executive branch cannot unilaterally withhold or cancel them without following legally prescribed procedures. Cancelling approved grants typically requires: submitting a formal rescission request to Congress, obtaining Congressional approval within a specified timeframe, and providing formal notice and appeal opportunities to affected grant recipients. These procedural requirements exist to ensure executive power does not encroach upon Congress's "power of the purse" — the Constitution's exclusive grant of appropriations authority to Congress. DOGE's bypassing of these procedures to directly cancel grants was one of the core legal bases for the judge's unconstitutionality ruling.

Where Are the Boundaries for AI in Government Decision-Making?

Assistive Tools Cannot Replace Human Decision-Makers

This case draws a clear red line for AI in public administration. AI tools can serve as assistive means — helping organize data, performing initial screening, and improving efficiency — but they should never be the sole basis for critical decisions. When it comes to matters involving civil rights and public fund allocation, professional human judgment, legal review, and due process are indispensable.

Large language models have well-known limitations:

1. Hallucination problems: They generate content that appears plausible but is factually incorrect
2. Contextual understanding gaps: Their grasp of complex policy contexts can be severely flawed
3. Inability to bear legal responsibility: No legal entity can be held accountable for an AI's erroneous judgments

Using such tools to determine the fate of hundreds of millions of dollars in grants is not merely a technical misuse — it represents a serious failure of governance.

In fact, prior to this case, the use of AI in government decision-making had already sparked multiple international controversies. One of the most notable was the Dutch tax authority's use of an algorithmic system for fraud detection in childcare benefit applications, which erroneously flagged tens of thousands of families as fraudsters. This triggered what became known as the "childcare benefits scandal," a major political crisis that ultimately led to the collective resignation of the Dutch cabinet. On the regulatory front, the Biden administration issued an executive order on AI safety in 2023, requiring federal agencies to conduct risk assessments and ensure transparency when deploying AI systems. The EU's AI Act classifies AI systems used by governments for public service decisions as "high-risk," requiring compliance with strict standards. The DOGE ruling effectively represents the first major constitutional-level response by the U.S. judicial system to a government agency's misuse of general-purpose AI tools — its significance may well extend far beyond U.S. borders.

What Far-Reaching Effects Will This Ruling Have?

This ruling may set an important precedent. As more government agencies explore AI applications, the court's decision sends a clear signal: AI-assisted decision-making must be embedded within legitimate administrative procedure frameworks, not replace them.

Government departments using AI tools in the future may need to meet stricter transparency and accountability requirements:

• Clearly define AI's role in the decision-making process — as a reference only, not a determining factor
• Maintain complete human review stages to ensure every major decision is vetted by qualified professionals
• Safeguard affected parties' right to information and right to appeal — the use of AI cannot be an excuse to skip legally mandated procedures

Conclusion: No Matter How Powerful AI Becomes, It Cannot Replace the Rule of Law

The significance of this case — DOGE's use of ChatGPT to review grants being ruled unconstitutional — extends far beyond the incident itself. It reveals an increasingly urgent question in the age of rapid AI proliferation: when powerful AI tools are readily accessible, how do we prevent their misuse?

Technological convenience cannot serve as an excuse to bypass legal procedures, and "letting AI decide" cannot substitute for responsible public governance. The judge's ruling reminds us that no matter how advanced artificial intelligence becomes, in major decisions involving the public interest, the rule of law and due process remain inviolable bottom lines.