The Uncomfortable Truth About AI Literacy Is That It Looks Like Work

I just read Tanmay Bansal’s article on Medium titled “Everyone Is ‘Learning AI,’ But Nobody Really Understands This One Thing,” and I agree with his diagnosis, even if the click-bait-y title gave me hives: Most of what passes for AI education is just interface fluency. People memorize prompt templates, copy-paste boilerplate from LangChain tutorials, and call themselves AI experts. Prompt Engineers. It’s a cargo cult and they’re learning to “operate a vending machine,” as Bansal puts it, without understanding the mechanics inside. He’s right about that, and it needed saying.

But his prescription is where it falls apart.

Bansal argues that real AI literacy means understanding that large language models navigate probability in high-dimensional geometric space—that knowledge is encoded as statistical structure, that meaning is location, that hallucination is just the model landing in the suburbs of truth instead of the city center. These are useful metaphors backed by real math, and he explains the intuitions clearly. Then he offers a code example using cosine similarity to measure semantic distance between an AI-generated answer and a source document, which would be compelling if the code didn’t generate random vectors and call it a demonstration. The example literally cannot do what he says it does. That’s a fairly critical failure if your thesis is that people need to move past surface-level understanding.

The deeper issue however isn’t the broken code. It’s the question his article raises and never answers: once you understand the math, what the hell do you do with it?

First, a quick primer on AI, because there’s been some confusion. These systems don’t store text and retrieve it. They aren’t databases. They don’t select and collage pieces of images together. When you ask a question, the model isn’t looking up an answer in a filing cabinet. When you generate an image, you don’t get an assemblage of stolen works. The model “learned” statistical relationships between words or visuals across enormous amounts of data, and it generates output by predicting what comes next based on those patterns. Every response is new. Every response is also a guess—albeit a very sophisticated guess—informed by billions of parameters encoding the statistical structure of human language, but a guess nonetheless. The output has no relationship to truth, because the model has no concept of truth. It has a concept of probability.

This is what Bansal gets right, and it matters: these systems don’t know anything. They navigate probability. When the navigation lands somewhere accurate, the output is useful. When it doesn’t, the output is wrong. The output reads exactly the same either way, and the model doesn’t know when it’s wrong, because it doesn’t know when it’s right. It’s always doing the same thing: generating the most statistically probable next token. Confidence is baked into the architecture. It’s not a signal. It’s the default.

So you don’t get obviously wrong answers, you get plausible answers delivered with confidence and ad-hoc justifications that sound authoritative—and might be completely wrong.

Confident bullshit, if you’ll forgive the French vernacular.

And that’s the part that should keep professionals and educators up at night.

I can follow Bansal’s argument about vector space and cosine similarity easily enough. I spent over twelve years at Microsoft as a Senior Software Development Engineer, including work on AI systems, and I’ve been writing software for twenty-five years. So I understand the architecture. I’ve read the papers. I’m also a professional novelist who’s spent the last year and then some experimenting with AI tools in creative writing, cataloging how and why they categorically fail at professional-grade prose generation, developmental and line editing, and other important tasks requiring taste, judgment, empathy, and actual understanding. I’m technical enough that Bansal’s prescription—learn the math—isn’t beyond me, and also I’m deep enough in a specific creative domain to know that the math is irrelevant to the failure modes I actually encounter.

The novel I’m currently working on is a second-world fantasy in a Tudor-adjacent setting. After receiving professional feedback from my developmental editor, I decided to test three different AI models and see what they’d say. Not that I don’t trust my editor (he’s taken the red pen to over a million words of my prose and there’s no one I trust my work with more), but simply because I was perversely curious if LLMs were up to the task. Grok naturally called it pulp and suggested I self-publish “if polished,” because Grok is functionally useless. No surprise there. Claude Sonnet 4.5, a much better model, nevertheless imposed workshop theory about structural unity (among other bad advice) that would have meant major rewrites (my editor explicitly ordered me not to rewrite anything). Anthropic’s flagship model, Claude Opus 4.6, fared somewhat better, but still demanded character agency changes that would have betrayed the premise, such as insisting a teenaged female traveling player accused of treason in a sixteenth-century analog should “drive the plot” through plucky protagonist willpower, as if power would work remotely that way for someone of her gender and social and legal position. Three models, three different sets of disastrously wrong notes, zero overlap with the actual problems my editor found. All delivered with full confidence and plausible rationales.

It was all a steaming load of happy horseshit, as my grandfather used to say.

In a separate line-editing experiment for a different story narrated first person, present tense by a traumatized fourteen-year-old girl, Claude flagged the line “this scar will be a permanent reminder of an ugly afternoon” and insisted I replace “permanent reminder” with “souvenir.” Why? Because the model saw a common word pairing, pattern matched it to “cliché,” and reached for something “more literary”—and arguably far more cliché—without bothering to model who was speaking and just how wrong it would sound.

Now, keep in mind advanced models like Opus are rather good at analyzing prose. They can write a passable English Lit paper deconstructing a given text, which, on the surface at least, sure looks like they’re understanding things like theme and characterization and narrative voice. Yet they can’t line-edit their way out of a paper bag. Seems paradoxical, right? When I asked Claude what in the eff-word was going on, it cheerfully self-diagnosed: asked to evaluate prose, it can recognize what a passage is doing, recognize character voice, etc. Asked to edit the same prose, even in the same context window, it activates a correction-seeking mode that overrides that recognition and optimizes toward a statistical mean of “good writing.” And the more distinctive your voice, the more damage it does. None of this is fixable with better prompts or context (Claude had detailed project knowledge including 100K+ words of manuscript text and explicit instructions). The failure is architectural. Worse, it will actively and pathologically invent problems that don’t exist, retroactively justifying them with authoritative and plausible explanations that sound extremely convincing if you lack a finely-tuned bullshit detector.

Also architectural.

Bansal’s answer, apparently, is to “learn the math.” Understand that AI models navigate probability in high-dimensional geometric space. Learn what cosine similarity is. Write eval scripts. As I mentioned earlier, he provides a code demonstration employing cosine similarity to measure whether an AI-generated answer is semantically grounded in source data, which is the centerpiece of his practical advice, the thing that separates his leet “AI thinker” from the wannabe “tool user.” But the code generates random vectors, not embeddings from an actual model. Just damned random numbers. And because “numpy.random.rand” produces values uniformly distributed between zero and one—all positive, no meaningful variation in direction—cosine similarity between any two such vectors in 1536-dimensional space converges to approximately 0.75. Which is his threshold for “grounded.” Long story short, his example will hover around “pass” regardless of input, for any pair of texts, about any subject, because it’s measuring absofreakinglutely nothing. It is an ironic demonstration of performative depth in exactly the format he’s accusing others of mistaking for… you got it, depth.

<facepalm.gif>

Regardless, even if the dud script didn’t look like something regurgitated out of ChatGPT and actually measured anything, you know, real, how does it, or understanding vector space, or any of his advice help with detecting confident-sounding bullshit in a given professional domain?

The real kicker though is his article ends with examples that should have completely rewritten his thesis. He warns about a legal summarizer that swaps “defendant” for “plaintiff” because those words occupy nearby coordinates in vector space. He warns about a medical bot that recommends a dosage that’s statistically probable but biologically fatal. These are genuinely frightening, but his prescription for the people facing them is the same prescription he gives everyone else: learn the math, write eval scripts, think in systems. But the lawyer who needs to catch that defendant/plaintiff swap doesn’t need to understand embedding geometry. They need to have read enough AI-generated legal summaries to know that’s a thing it does. The doctor doesn’t need cosine similarity. They need to have tested the system against known cases enough times to map where it careens off into medical malpractice territory. A novelist doesn’t need vector math to catch a model destroying a fourteen-year-old’s voice with a French loanword. They need enough craft experience to hear the wrongness.

Scripts calculating cosines aren’t going to help with those issues.

Bansal identifies domain-specific failures as the real danger and then offers a domain-agnostic solution, and that’s the contradiction at the heart of his piece. The math can tell you why “defendant” and “plaintiff” are close in vector space, should you be perversely curious, but it can’t warn you they got swapped in a given output. Only domain expertise can do that—the kind that comes from years of practice in a specific field, not from a weekend course on embeddings (or a Medium subscription).

The Bansal critique matters not because he’s personally wrong about math, but because the wrong prescription has real casualties, and they’re concentrated among the people who can least afford them. When an experienced lawyer reads an AI-generated brief and something feels wrong about a citation, they cross-check, and spot the hallucination. They have the instinct because they’ve read thousands of real briefs. A law student reads the same output and it looks like every other brief they’ve encountered in their short career. They don’t have the pattern library to intuit something’s off.

An experienced novelist hears “souvenir” and knows instantly it’s catastrophic for a given slum-kid’s voice. An aspiring writer gets “replace this cliché with something more literary,” and it sounds like exactly the kind of feedback they’ve been told to seek. It comes wrapped in craft language. It references real principles. It sounds like a good editor having a good day. They take the note, and their narrator’s voice breaks, and they don’t know why the scene stopped working.

AI is most dangerous precisely where it’s being marketed hardest—as a learning aid, a writing coach, a study partner, a tutor. The person who most needs reliable feedback is the person least equipped to evaluate whether the feedback is reliable. “Learn vector math” does nothing for a first-year law student trying to figure out if the AI just hallucinated a precedent. It does nothing for an aspiring novelist trying to figure out if the AI just taught them something true about craft or vomited up plausible horseshit with confidence and authority.

The entire discourse around AI education is modeled on traditional technical education: learn the fundamentals, build up from there, acquire a body of knowledge. Bansal’s article is a perfect example. He’s built a hierarchy with prompt engineers at the bottom and math-understanders at the top, and the implicit promise is that climbing the ladder makes you competent. But the ladder is leaning against the wrong wall. The failure modes that matter in practice aren’t derivable from first principles. You can’t predict from the architecture that Claude will swap a perfectly fine commonly-used term out of the mouth of a teenaged orphan raised in the slums and replace it with a tired literary-sounding cliché, or that Grok will miscategorize upmarket fantasy as pulp, or that a legal summation will silently swap defendant for plaintiff—or that the confidently delivered dosage recommendation will, in fact, kill your patient (oopsies!). These failures are domain-specific, context-dependent, and discoverable only through use. Repeated, unglamorous, empirical use, running the tool against your own work in your own field and paying attention to where it breaks.

That’s not a body of knowledge. It’s a practice. And the real hierarchy isn’t prompt engineers at the bottom and architecture-understanders at the top. It’s people who’ve done the hours in their own domain versus people who haven’t. The math is nice to know, potentially helpful, but ultimately optional. The hours aren’t.

Nobody’s teaching this because there’s nothing to package. You can’t build a “Master AI Literacy in 4 Hours” course around “go use the tool on real work in your field for a year and keep notes on where it fails.” You certainly won’t sell any membership subscriptions. There’s no curriculum. There’s no certificate. There’s no code example you can put in a Medium article to make it look like you’ve arrived. There’s just the work, and “do the [fucking] work” never sells. Trust me, I’ve tried. No one likes my advice that to be a good writer you need to burn through hundreds of thousands of shit words—a million is a decent benchmark—and then, probably, you’ll be ready to actually start writing for reals.

Bansal isn’t wrong about the instinct. The impulse to go deeper, to move past surface-level engagement with these tools and understand something real about what they are and what they’re doing, is the right impulse. Most people are learning a UI and calling it expertise, and that’s a problem worth naming. I’ll give him credit for that.

But where he stops is one layer short. Understanding the engine is a helpful step, not a destination. The destination is the specific, unglamorous knowledge you can only build by working with the tool in your own domain long enough to map its failure modes, and that knowledge doesn’t generalize. What I’ve learned about how AI fails at creative writing is useless to a lawyer. What a lawyer learns about hallucinated legal precedent is useless to a urologist. While the math is exactly the same in every case, the failures aren’t.

The uncomfortable truth about AI literacy is that it looks like the kind of expertise we already know how to build but have never been able to shortcut. The kind that comes from years of doing the work, making the mistakes, and developing the judgment to know when something that sounds right is wrong. We’ve always known there’s no substitute for that. Everyone just keeps saying this time will be different.

It won’t be.