What Should Students Be Taught on Using AI?

28 Sep, 2026

The Prompting Delusion

Walk into a classroom running an "AI literacy" module today and there is a good chance the lesson is about prompt engineering: how to phrase a request, how to add "role" and "context" tags, how to chain instructions for a better output. It is a reasonable thing to teach in 2023. It is a strange thing to still be teaching as the primary skill in 2026, because the models have gotten good enough that clever phrasing is rapidly losing its value. A student who spends a semester learning prompt tricks is investing in a skill with a short shelf life.

The real problem is not that students prompt poorly. It's that most of them use AI tools daily without understanding what the tool actually is, where their words go after they hit enter, whether the output in front of them is true, or what they lose by using it. That is a much bigger gap than prompting, and it is the one worth closing first.

This post argues that AI literacy is not about learning to talk to a machine. It is about learning to think critically about the machine itself — who owns it, what it does with your data, how transparent its construction is, and what its fundamental limitations are. Five areas matter more than prompt tutorials: data privacy, the difference between closed, open-weight, and open-source models, the "compiled binary" problem hidden inside that distinction, the probabilistic (not factual) nature of how these systems generate text, and the risk of cognitive offloading.

Data Privacy: You Are Feeding the Machine

The illusion of a private conversation

Students tend to treat a chatbot the way they'd treat a search bar or a private notebook. It isn't either. Every prompt is a data transaction: text leaves the student's device, travels to a company's servers, and may be stored, reviewed, or used to improve a commercial product. Nothing about the conversational interface implies privacy by default — that has to be established by policy, not assumed from tone.

Questions students should learn to ask before using any AI tool

• Does this tool use my prompts to train future models, and can I opt out?
• How long is my data retained — is there a genuine deletion option, or only a "hidden from you" option?
• Is there any right to have my data permanently erased?
• Is my data shared with advertisers, affiliates, or other third parties?

The "free tier" trade-off

If a tool is free and commercially operated, the economics have to work somehow. Often that means the student's usage — their prompts, their patterns, their corrections — is the resource that subsidizes free access. This isn't a criticism of any particular company; it's simply how many consumer software businesses are financed, and students deserve to understand the business model behind a tool before handing it their schoolwork, their private worries, or their identifying details.

Reading the privacy policy

Few students have ever opened a privacy policy or terms-of-service document, and fewer still know what to look for. A practical classroom exercise: have students pull up the privacy policies of two or three widely used AI tools side by side and locate four specific clauses — data retention period, whether prompts are used for model training, third-party data sharing, and the legal jurisdiction governing disputes. The exercise is less about memorizing legal language and more about building the habit of checking before trusting.

Legal and institutional compliance

Frameworks like FERPA and COPPA in the United States, and GDPR in the European Union, exist specifically to constrain how minors' data can be collected and used. Many AI tools marketed directly to students or teachers were not built with these frameworks in mind, and schools that adopt a tool without checking its compliance status are taking on a duty-of-care risk on behalf of children who never consented to anything. This is one of the concerns that has driven institutional caution elsewhere — New York City's education department, for instance, cited data governance for minors as a central reason for pausing generative AI tools in grades K–8 this year.

Closed Source, Open Weights, and True Open Source

Why the label "open" is doing too much work

"AI" is often discussed as if it were one category of thing, but the label hides enormous differences in transparency, control, and what a user is actually allowed to do with the software. Students deserve to know which category the tool in front of them falls into.

Closed source models — the architecture, training code, training data, and final weights are all proprietary and undisclosed. Most of the widely used commercial assistants fall into this bucket. There is no way for an outside researcher to audit what's inside; users are relying entirely on the vendor's own claims about safety and behavior.

Open-weight models, sometimes loosely marketed as "open source," release the trained parameters — the billions of numbers that make the model run — and often the architecture that describes the network's shape. What they typically withhold is the training code and, almost universally, the training dataset itself. Many also attach licenses with real restrictions. Meta's Llama 3 license, for example, permits broad commercial use but requires any company whose products reach more than 700 million monthly active users to obtain a separate license directly from Meta — a condition that has nothing to do with the freedoms typically associated with open-source software.

True open-source AI, as defined by the Open Source Initiative's Open Source AI Definition (OSAID 1.0), requires that a system grant users the freedom to use, study, modify, and share it, and that this be backed by access to the model's source code, its parameters, and sufficiently detailed information about its training data — enough that a skilled person could substantially recreate the system. This standard does not always require the raw dataset to be redistributed in full (data can be legally restricted), but it does require open code, clear disclosure of data provenance, and a genuine ability to inspect and rebuild the system. Only models meeting this bar — projects like OLMo or Pythia — allow a third party to actually audit and reproduce what they're using.

A simple comparison for the classroom:

The "Compiled Binary" Problem

A useful analogy for students who have taken any introductory programming class: think of the released model weights as a compiled binary. You can run it, you can even redistribute it, but you cannot read the logic that produced it just by looking at the numbers. The architecture and training code are the source code — the actual instructions that define how the system learns. The training dataset is the build environment — without it, nobody outside the company that made the model can verify what went into it or reproduce it from scratch.

This framing explains why an "open-weight" release is not the same as an open-source release. You can inspect a binary's outputs, but you cannot inspect its internal reasoning, its hidden behaviors, or the influence of specific training data on specific biases. You also lose the benefit that traditional software security has relied on for decades: reproducible builds, where anyone can compile the published source and check that it produces the exact binary being distributed. Without published training data and training code, nobody can verify that a model's weights are actually the honest output of the process the vendor describes — trust has to be extended to the vendor rather than verified independently.

That distinction is worth teaching explicitly: an open-weight model asks the user to trust a company; a true open-source model lets the user verify a process. Both can be useful, but they are not interchangeable, and a marketing label of "open" should not be taken as a guarantee of either transparency or unrestricted freedom to use, modify, or redistribute the model — as the Llama license's usage cap illustrates.

Critical Thinking: Probabilistic, Not Factual

How generation actually works

A large language model does not look facts up in a database and report them back. It predicts the statistically most likely next word given the patterns in its training data. This is precisely why it can produce a confident, fluent, well-structured paragraph that is also factually wrong — fluency and accuracy are not the same property, and the model is optimized for the former.

Hallucination is architectural, not accidental

Fabricated citations, invented statistics, and confidently wrong dates are not bugs to be patched away entirely; they are a predictable consequence of how these systems generate text. Students should be taught to treat every AI output as a first draft to verify, never as a finished answer to submit.

Bias hides where transparency is absent

Whatever perspectives dominate a training dataset will shape a model's defaults, and when the dataset is undisclosed — as it is for both closed-source and most open-weight models — those defaults are invisible until someone stumbles across them. A fair question to teach students to ask of any AI output: whose perspective is this answer built on, and what viewpoint might be missing?

A verification habit worth building

1. What exactly is the claim being made?
2. Can this be checked against a primary source?
3. Did the AI cite anything — and does that citation actually exist when checked?
4. Am I using this tool to learn something, or to skip the process of learning it?

Cognitive Offloading: Augmentation vs. Automation

There is now direct evidence that how a student uses an AI tool changes what they retain. A 2025 MIT Media Lab study divided participants into three groups writing essays — one using an LLM, one using a search engine, and one writing unaided — and tracked their brain activity with EEG over four months. The unaided group showed the strongest and most widely distributed neural connectivity; the LLM group showed the weakest, along with the lowest self-reported sense of ownership over their own essays and the poorest ability to recall or quote what they had supposedly written. The researchers described the accumulated effect of repeatedly outsourcing this cognitive work as "cognitive debt."

That finding maps onto a distinction worth teaching directly: augmentation is using AI to brainstorm, stress-test an argument, explain a stuck concept, or debug code, with the student still doing the core thinking. Automation is using AI to generate the finished product with minimal input, in which case the student stops being the author and becomes a filter passing text through. The skill that was supposed to be built — writing, reasoning, working through a hard problem — never gets exercised, and evidence suggests that skipping the struggle has a measurable cost.

A simple self-audit worth teaching before any AI-assisted work gets submitted: did I learn something in this process, or did I just save time? The goal of an assignment is the thinking it's meant to build, not merely the document at the end of it.

Practical Takeaways for Educators

Shift the curriculum. Spend less time on prompt phrasing and more on data literacy, model transparency, and habits of verification — skills that don't expire when the interface changes.

Classroom exercises worth running:

• A privacy-policy comparison across two or three AI tools, focused on retention, training use, and third-party sharing.
• A side-by-side look at a genuinely open-source model against a popular open-weight one, showing concretely what is and isn't disclosed.
• A "hallucination hunt," where students are given AI-generated text seeded with deliberate errors and asked to fact-check every claim.
• A binary-analogy lab: show students a compiled program next to its source code, then show model weights next to training code, and let the comparison make the abstraction concrete.

Policy recommendations for schools:

• Vet any AI tool for student data compliance before adoption, rather than after a problem surfaces.
• Favor tools with clear, genuine data-deletion options over ones that merely hide data from the user's view.
• Where practical, favor tools whose architecture and data practices are actually inspectable, rather than ones that merely carry the word "open" in their marketing.
• Teach the reasoning behind each guideline, not just the rule itself — students who understand why a practice matters are more likely to apply it once the specific tool or policy changes.

Teach Them to Question the Machine

AI is not magic. It is software: built by specific people, trained on specific data, licensed under specific terms, and shaped by specific economic incentives. None of that is disqualifying — plenty of useful, trustworthy tools are built this way — but none of it should be invisible to the person using it every day.

Students are not taught to accept a textbook's claims without question; the whole point of teaching source evaluation is to build the habit of checking who wrote something, why, and what evidence backs it. There is no principled reason to hold a chatbot to a lower standard than a textbook simply because its answers arrive faster and sound more confident.

The goal of AI education is not to produce students who can use these tools efficiently. Efficiency comes on its own, quickly, without much instruction. The goal is to produce people who can use them critically — who ask where the data goes, who know the difference between a system they can verify and one they can only trust, and who can tell the difference between a tool that helped them think and one that thought for them.