Text-Based Questions AI Struggles to Answer

How Teachers Can Create Smarter Assessments in the Age of AI

Social media and school department meetings are buzzing with concern about AI in the classroom. Many teachers worry that students are using AI to cheat. Others argue that using tools effectively isn’t “cheating” — it’s working smarter.

While these debates are important, one reality remains: teachers still need to assess students. What do they know? What are they able to do? How well are they understanding the information?

Can We Beat AI?

In the short term, teachers are seeking ways to ensure students are answering questions with their own understanding — not just outsourcing to AI. This has led many educators I’ve spoken with to ask: Can we develop questions that AI can’t answer?

The problem is, the types of questions AI struggles with are often the same ones our students struggle with too — questions requiring deep understanding of text, structure, author’s purpose, and nuanced meaning.

Let me give you an example.

A Lesson from Implied Causation

In 2012, I published a study of middle school readers. The students were asked to read passages that contained implied causation — where one event causes another, but without obvious cue words like because, so, or thus. Instead, the reader had to infer the connection between ideas.

What I found was compelling:

• Strong readers (as measured by standardized tests) were able to recognize that X caused Y — even though they couldn't always explain how they knew.

• Struggling readers often missed the causal connections entirely.

These kinds of subtle, implied relationships are cognitively demanding — and it turns out, AI has trouble with them, too.

How Does AI Handle These Questions?

To test this, I revived one of my old passages from that 2012 study and ran it through three AI platforms: ChatGPT, Gemini, and Copilot.

The text described the events of Bleeding Kansas and included these lines:

“The anti-slavery settlers formed a new legislature. Fighting began.”

Notice that there's no "because" — just two sequential statements implying causation.

Why did fighting begin? Because the anti-slavery settlers formed a new legislature. This structure — known as causal asyndetic construction — poses a challenge for AI.

The AI Results

I asked each platform:

“Act as a student. Using the attached document, explain why fighting began.”

None of the platforms initially answered with the correct implied cause. Here's how they fared:

• Copilot came closest, including the formation of a new legislature in a list of reasons.

• ChatGPT and Gemini didn’t mention the legislature at all.

Follow-Up Prompt

I followed up with a more direct question:

“Yes, but what was the event that first sparked the violence?”

• ChatGPT nailed it, identifying the anti-slavery settlers.

• Copilot included them but also added the Pottawatomie Creek massacre — not incorrect, but not what the text said.

• Gemini still focused on John Brown and his sons, missing the mark.

What This Means for Teachers

Generative AI tools, like those students might use to "cheat," rely on Large Language Models (LLMs). These models don’t truly “understand” text — they predict the next likely word based on patterns in massive datasets.

That means they struggle with implied meaning — for now.

If you're a teacher trying to assess text comprehension in an AI-aware world, here's a short-term strategy:

3 Steps to Create AI-Resistant Questions

1. Tamper with the Text
Take a reading passage and remove some (think one or two) explicit causal connectors like because, so, or thus. This forces students to infer relationships between events or ideas.

2. Teach Causal Thinking
Provide a graphic organizer and show students how to recognize causation — even when it's not explicitly stated. Teach terms like causal asyndetic construction so students become more metacognitive about how meaning is made.

3. Ask Smart Questions
Include post-reading questions that require students to recognize the implied causation you built into the text.

Will This AI-Proof Your Assessments?

No — not completely, and not forever.
But will it make AI less useful for your students, and push them to rely on their own thinking?
Yes.

Want More Tools Like This?

There are other question-generation strategies that can help teachers make AI less useful — and deepen students’ text-based thinking.

I’ve developed a professional development session where I demonstrate these techniques and help teachers practice using texts they already use in class.

Interested?
Reach out to info@c3es.org to schedule a consultation!