Why AI Search Needs Intent (and Why DITA XML Makes It Possible)

Search used to be about matching keywords. Today, AI-powered search engines promise to “understand what users mean.” In practice, they struggle with something far more basic: intent.

Intent is the goal behind a search query—the job the user is trying to get done at that moment. Are they trying to learn, do, fix, compare, or verify? Intent answers why the question was asked, not just what words were typed.

For technical content, intent matters more than phrasing. A query like “configure OAuth token refresh” can signal at least four different needs:

• a conceptual explanation,

• a step-by-step task,

• a troubleshooting scenario,

• or a confirmation that a system even supports the feature.

The words don’t change. The intent does.

Why AI Search Engines Struggle With Intent

AI search systems are trained on language patterns, not user goals. That creates several hard problems:

• Queries are underspecified
  Users compress complex situations into a few words (often not even in a sentence). AI must infer missing context we fail to provide.

• Intent evolves across the customer journey
  The same query appears during learning, implementation, and troubleshooting phases of the journey.

• Unstructured content hides purpose
  When documentation mixes concepts, tasks, and reference material together, AI has no reliable signal to determine which kind of answer fits the situation.

When intent is unclear, AI systems fall back on probability. That’s when you get answers that sound right, rank well, and still fail the user.

Intent is the reason behind a search query. It answers the question: What is the person actually trying to accomplish right now?

• Are they trying to learn a concept, complete a task, fix a problem, compare options, or confirm a decision they’ve already made?

The challenge is that intent rarely appears explicitly in the query.

A search like “reset authentication token” could mean:

• “I need step-by-step instructions right now.”

• “I want to understand how token-based authentication works.”

• “I’m evaluating whether this system supports token rotation.”

• “I already tried something and it failed—what did I miss?”

The words stay the same. The intent changes completely.

Why AI Has Trouble Inferring Intent

Large language models are good at generating plausible answers, but intent inference is not a text-generation problem. It’s a context problem.

AI search systems struggle because:

1. Queries are compressed signals
   Users type the shortest thing that might work, not a full explanation of their situation. AI must guess what’s missing.

2. Intent shifts mid-journey
   A user may start by learning, then move to doing, then to troubleshooting (all with nearly identical queries).

3. Training data reflects language, not goals
   LLMs learn patterns of words, not the underlying job the user is trying to get done.
   
   👉🏾 Related: MIT researchers find a shortcoming that makes LLMs less reliable

4. Documentation rarely exposes intent explicitly
   Most docs are organized by product structure, not by user intent, task maturity, or decision state.

When intent is unclear, AI fills the gap with probability. That’s where hallucinations, irrelevant answers, and dangerously confident responses come from.