The Boundaries of AI: What Questions Can It Never Touch?

An Overlooked Question

As AI continues to make breakthroughs in areas like Go, protein folding, and code generation, a more fundamental question surfaces: Is every problem within AI's reach? A discussion thread on Hacker News raised this question. Although it received only 4 points and no comments, the issue it points to deserves serious attention from every technology professional.

The core of this topic is not to disparage AI's capabilities, but to clarify its boundaries—which problems are suitable to hand over to machines, and which problems are defined by their very nature as ones that humans must face.

Computable and Non-Computable: The Theoretical Hard Boundary

To discuss the boundaries of AI's capabilities, one cannot avoid the basic conclusions of computation theory. The Turing machine model tells us that there exists a class of problems that are non-computable—no algorithm can produce a definite answer within a finite number of steps. The halting problem is the most classic example; it mathematically proves that a general-purpose solver is impossible.

This means that no matter how AI architectures evolve or how much computing power expands, as long as AI is still built on computability, it cannot cross this theoretical red line. No matter how powerful GPT-style models become, they are still essentially performing definable function mappings. When the solution to a problem is itself non-computable, model scale is not the answer.

It should be noted that this boundary does not negate AI's practical value. Most real-world problems fall within the computable range, and AI's progress in these areas is real. But inferring from "AI can solve many problems" to "AI can solve all problems" is logically invalid.

The halting problem is an undecidable problem proposed by Alan Turing in 1936: given a program and its input, there is no general algorithm that can determine within a finite number of steps whether the program will eventually halt or run forever. Using the diagonal method, Turing proved that if such a decision procedure were assumed to exist, constructing a self-referential counterexample would lead to a contradiction. The deeper implication of this conclusion is that non-computability does not arise from insufficient computing power or algorithms that are not clever enough, but from a fundamental limitation in logic. Gödel's incompleteness theorems echo this, together delimiting the expressive boundaries of formal systems. For AI, this means that any system based on the Turing machine model cannot become a universal solver—the unsolvability of certain problems is embedded in the very concept of computation, not an engineering problem.

Practical Boundaries Beyond Undecidability

In addition to purely theoretical non-computability, several other types of problems pose structural challenges to AI:

Value judgment problems. Ethical dilemmas, policy trade-offs, and fairness in resource allocation—the answers to these questions depend on human choices rather than objective facts. AI can list pros and cons and simulate consequences, but the final authority on "what should be done" cannot be automatically derived from data.

Open-ended problems. Major breakthroughs in scientific research often come from redefining the problem itself, rather than solving within an established framework. AI excels at optimizing under a given objective, but the ability to "ask a good question" is still difficult to train in a formalized way.

Problems requiring embodied experience. When it comes to fine-grained operations in the physical world and subtle judgments in real social situations, AI lacks direct interactive experience with the world; its "understanding" is indirect and statistical.

According to the direction of the Hacker News thread (though no comments elaborated on it), such questioning usually revolves around "whether AI possesses true understanding," and the gap between understanding and computation is precisely the focus of the debate.

Another Interpretation: The Boundary Is Moving

There is also a strong rebuttal to the above arguments: the set of so-called "problems AI cannot touch" has been shrinking throughout history.

Translation, driving, and image recognition—once considered tasks only humans could perform—have now been partially or largely taken over by AI. From this perspective, although today's "non-computability" argument holds, its actual scope of impact may be far smaller than the theory suggests. Many tasks classified as "requiring understanding" are, under finer-grained analysis, merely complex pattern matching.

The disagreement between the two positions lies in: Are AI's limitations essential, or are they a temporary phenomenon of the current stage of technology? There is currently no consensus answer. Essentialists cite computation theory, while gradualists cite the history of technology. Both arguments have their rationality and their blind spots.

Moravec's paradox provides an intriguing footnote to this boundary debate: contrary to intuition, advanced reasoning tasks (such as chess and mathematical proofs) are relatively easy for computers, while the perceptual and motor skills possessed by human toddlers (such as walking, recognizing faces, and understanding common sense in natural language) are extremely difficult to formalize. This paradox was proposed by Hans Moravec and others in the 1980s, revealing the inversion of "difficult" and "easy" from the perspectives of humans and machines. It both supports gradualists—the boundary is indeed moving, just in unexpected directions—and reminds essentialists—behind some seemingly simple tasks lie deep capabilities that have not been formalized. The recent breakthroughs of large models in language tasks have not dissolved this paradox; instead, they have made it more prominent in the field of embodied intelligence.

Practical Implications for Practitioners

Rather than waiting for this debate to settle, it is better to turn it into actionable questions:

When designing AI systems, clearly distinguish between two types of tasks: "optimizing a known objective" and "discovering a new objective." The former is suitable to hand over to models; the latter still requires human judgment. When evaluating AI solutions, beware of equating quantifiable with important and non-quantifiable with unimportant. When discussing AI capabilities, distinguish between "performing well on benchmarks" and "being reliable in open environments."

The boundary of AI is not a question that needs a once-and-for-all answer, but a coordinate system that requires continuous calibration. Understanding which problems are within range and which are out of range is more valuable than simply pursuing an expansion of range.

One-line Summary

AI is not omnipotent: computation theory sets hard boundaries, but whether those boundaries can move remains an open debate.

Paragraph Summary

Starting from a Hacker News discussion, the article asks whether there are problems AI cannot touch. It first invokes computation theory, pointing out that non-computable problems such as the halting problem constitute a theoretical hard boundary for AI that cannot be crossed no matter how computing power grows. It then analyzes three types of structural challenges: value judgments, open-ended problems, and embodied experience. Next, it presents the rebuttal: historically, AI's boundaries have continuously moved, and many tasks once thought to be exclusively human have been taken over; the limitations may be only temporary. Finally, it offers actionable suggestions for practitioners: distinguish between optimizing known objectives and discovering new ones, beware of quantification bias, and continuously calibrate the understanding of AI's capability boundaries.