In the fall of 2023, researchers gave nearly a thousand high school math students in Turkey access to GPT-4 while they practiced. One group used a standard ChatGPT-style tool. Another used a tutor designed to give hints instead of answers.
During practice, the AI looked like a success. Scores rose 48% with the standard tool and 127% with the tutor.
Then the researchers took the AI away. On the exam, students who had used the standard tool scored 17% worse than students who never had AI at all.
They got more answers right, but they learned less. And checking whether answers are correct, the way we usually measure learning, would have missed it. (Bastani et al., PNAS, 2025)
This was one study of math practice at one school, but the pattern is worth taking seriously.
A system built for another time
Traditional education follows a familiar pattern: teachers explain, students study, and tests measure what they remember. That made sense when information was hard to find and schools needed to teach knowledge at scale. Knowledge still matters. But today information is instant, and AI can solve problems and write essays in seconds.
If a student can produce a convincing essay without understanding it, what does the essay tell us? These aren’t just questions about cheating. They’re questions about what education is supposed to measure. When answers become easy to produce, the thinking behind them is what’s worth teaching.
What that looks like in practice
We should pay more attention to how students reach their answers. Can they explain their reasoning? Notice when something doesn’t add up? Apply an idea somewhere new? This can be as simple as:
Asking students to explain a solution before moving on Grading the reasoning, not just the answer Having students find the mistake in a flawed answer (an AI-generated one works well) Following a familiar problem with one in an unfamiliar setting
The study points the same way. For students who used the hint-based tutor, the negative effects on learning were largely mitigated. The problem wasn’t AI itself. It was AI doing the thinking for the student.
What are we preparing students for?
We don’t need to know exactly what the future holds to know students will need more than recall. They’ll need the judgment to evaluate answers, the confidence to question them, and the ability to use knowledge in situations they’ve never seen.
If answers are becoming easier to find, let’s make sure we’re still teaching students how to think.