Why Vibe Coding Policy Questions Is a Bad Idea (Just Ask the Swedish PM)
Just a few weeks ago, Sweden’s Prime Minister Ulf Kristersson faced criticism after admitting he regularly consults ChatGPT for “second opinions” on policy matters. Tech experts condemned the practice, citing security risks—sensitive information uploaded to commercial AI platforms, potential data breaches, lack of oversight. Fair points. Sophisticated alternatives exist: classified AI systems like Palantir AIP, purpose-built government tools with proper security protocols.
Yet even if these security concerns were resolved, something would still feel wrong about a prime minister consulting AI for policy guidance. But why does this feel troubling? The unease runs deeper than cybersecurity or operational risk.
The Virtue Problem
Political leadership demands intellectual virtues: sound judgment, intellectual honesty, the ability to weigh competing interests, courage to make difficult decisions and accept responsibility for consequences. When Kristersson consults ChatGPT for policy advice, the concern goes beyond using the wrong tool—it’s about potentially substituting artificial responses for genuine political thinking.
The Intentional Stance Trap
When we interact with sophisticated AI systems, we cannot help but apply what philosopher Daniel Dennett calls the “intentional stance”—treating them as if they have beliefs, desires, and coherent reasoning processes. This cognitive approach, which involves attributing mental states to complex systems to predict their behavior, works remarkably well even when we know the system lacks genuine mental states.
But when we apply the intentional stance to AI systems in consultation contexts, we implicitly assume more than just predictable behavior. We treat them as epistemic agents—entities whose “beliefs” are formed through reasoning about evidence and whose “judgments” reflect some form of intellectual character. When Kristersson seeks a “second opinion” from ChatGPT, he isn’t just predicting its textual output; he’s treating it as if it possesses the epistemic virtues that make opinions worth seeking: careful reasoning, appropriate uncertainty, commitment to accuracy over mere helpfulness.
This assumption is natural because, in human interactions, treating someone as an intentional agent whose opinions matter necessarily implies that they exercise intellectual virtues. We consult advisors precisely because we believe their beliefs and judgments emerge from virtuous thinking processes—that they care about truth, acknowledge uncertainty appropriately, and reason carefully about evidence.
Virtue epistemology—the philosophical study of what makes someone a good thinker—focuses on intellectual character traits like intellectual honesty (acknowledging uncertainty and the limits of one’s knowledge), intellectual courage (willingness to challenge received wisdom when evidence demands it), appropriate confidence calibration (expressing certainty proportional to the strength of available evidence), and epistemic empathy (fairly representing opposing viewpoints).
These virtues work together to create what Aristotelian virtue theorists call integrated intellectual character. We expect epistemic agents not just to provide information, but to care about truth in the right way—to be motivated by accuracy rather than mere helpfulness, to resist overstatement, to maintain coherent standards of evidence.
What LLMs Actually Do
Large language models are trained on vast datasets scraped from the internet with minimal curation for epistemic quality. Academic papers, opinion pieces, conspiracy theories, and marketing copy all contribute equally to the statistical patterns these systems learn. The training objective is not truth-seeking but prediction: given a prompt, what text is most likely to follow?
This approach treats all text as equally valid training material. A peer-reviewed study carries the same statistical weight as a blog post, provided both fit the patterns the model is learning. The system becomes skilled at mimicking authoritative discourse without any underlying commitment to accuracy, appropriate uncertainty, or careful reasoning.
LLMs exhibit what we might call intellectual vices: overconfidence (confidently discussing topics beyond their knowledge), inconsistency (giving different answers to the same question asked different ways), and indifference to truth (optimizing for seeming helpful rather than being accurate). These aren’t bugs—they’re natural consequences of the training process.
When minimal guardrails are applied through techniques like reinforcement learning from human feedback, they typically focus on preventing obviously harmful outputs rather than instilling systematic epistemic virtues. The result is systems that can confidently discuss complex topics while lacking the fundamental commitment to truth that characterizes genuine expertise.
The Consistency Problem
This manifests clearly in AI systems’ notorious inconsistency. Ask the same strategic question with slightly different wording, and you may receive substantially different answers. This variability reveals that what appears to be stable beliefs and careful reasoning is actually contextual pattern-matching.
For human advisors, such inconsistency would signal unreliability or insufficient expertise. But AI systems’ inconsistency reflects their fundamental nature: they are not repositories of stable beliefs but dynamic pattern-matching engines that generate responses based on immediate context rather than persistent understanding.
When Kristersson consults ChatGPT repeatedly on related policy questions, he may unknowingly receive different “opinions” that reflect statistical variations rather than evolving judgment. The system cannot maintain the kind of coherent, evidence-based perspective that genuine expertise requires.
Using AI Responsibly
None of this means AI tools are worthless for governance. They can be valuable aids to human thinking—helping explore possibilities, identify blind spots, synthesize information from large datasets.
Consider programming, where AI assistance has become commonplace. Developers use LLMs to generate code snippets, explore different approaches to problems, and debug complex systems. But the process includes crucial verification: the code must compile, pass tests, and actually solve the intended problem. The programmer retains responsibility for understanding the solution and ensuring it works.
Peter Naur’s influential essay “Programming as Theory Building” argued that programming is fundamentally about developing a mental model of the problem and its solution. When programmers use LLMs responsibly, they’re not outsourcing this theory-building process—they’re using AI to rapidly explore different theoretical approaches. The LLM suggests possibilities; the programmer evaluates, tests, and either accepts or rejects them. The result is both working code and, ideally, enhanced understanding of the problem space.
But programming also illustrates how things can go wrong. “Vibe coding”—using LLMs to generate code without understanding what it does—has become increasingly common. Developers copy-paste AI-generated solutions that seem to work without grasping the underlying logic, dependencies, or potential failure modes. This violates fundamental epistemic virtues: intellectual honesty (acknowledging the limits of one’s understanding), intellectual humility (recognizing when one doesn’t comprehend a solution), and epistemic responsibility (ensuring one can maintain and modify what one builds).
When Kristersson consults ChatGPT, the concern isn’t just that he’s using the wrong tool—it’s that he’s treating any AI system as a source of genuine strategic wisdom rather than sophisticated pattern-matching. Leadership requires intellectual virtues that cannot be automated: the wisdom to recognize the limits of one’s knowledge, the courage to act despite uncertainty, and the integrity to accept responsibility for decisions. We are all imperfect—we fall prey to overconfidence, rely too heavily on questionable sources, mistake authority for accuracy. But recognizing these failings is what intellectual humility demands. When we treat AI systems as genuine advisors rather than text generators, we risk abandoning the very thinking that good judgment requires.
Image source
Muszka, Dániel, and Antal Münnich. “Gondolkodó gépek.” (Thinking Machines) Budapest: MDV, 1960. OSA Archivum. http://diafilm.osaarchivum.org/public/?fs=3597
Leave a Reply