What Is Query Rewriting? Better Questions for Better Retrieval

Users write queries the way they think: short, colloquial, ambiguous, referencing earlier turns, mixing several questions in one. Documents are written differently. Query rewriting closes the gap by transforming the raw query before retrieval: clarifying, expanding, decomposing, contextualizing, or generating a hypothetical answer to search with. Done well, it lifts recall and precision; done carelessly, it adds latency and drifts from intent. This explainer covers the techniques and when to use them, drawing on FISTA Solutions' AI enablement practice. The retrieval design it feeds is in what is hybrid search and the system context in what is rag.

What is query rewriting?

Query rewriting is a preprocessing stage in which the user's input is converted into one or more retrieval queries, typically by a language model or lightweight rules, before the search index is queried. The rewritten queries may include the original, and results are merged. The goal is to search for what the user means using the vocabulary and structure the index responds to. Search foundations are in what is semantic search.

What techniques does query rewriting include?

Technique	What it does	Best for
Clarification	Rephrases ambiguous or terse queries into clear intent	Short, informal queries
Expansion	Adds synonyms, abbreviations, and domain terms	Vocabulary mismatch, lexical search
Decomposition	Splits multi-part questions into sub-queries	Comparative and compound questions
Conversational rewriting	Resolves references using prior turns	Chat interfaces
Hypothetical document embedding	Generates a plausible answer and embeds it for retrieval	Questions phrased unlike documents
Multi-query generation	Produces several phrasings and merges results	Recall-critical systems
Filter extraction	Pulls dates, entities, and types into metadata filters	Structured corpora
Step-back prompting	Forms a broader question to retrieve background first	Reasoning-heavy questions

How does expansion help lexical search?

Keyword retrievers match tokens, so a query for "bill" misses documents that say "invoice." Expansion with synonyms, abbreviations, and domain vocabulary, drawn from a model or a curated thesaurus, lets lexical search match what documents actually say. Lexical scoring is in what is bm25.

How does decomposition handle complex questions?

A question comparing two policies or asking for several facts rarely has a single passage that answers it. Decomposition produces sub-queries, each retrieves its evidence, and the model synthesizes across them with citations. Agentic retrieval extends this with iterative searching; see how to build an agentic rag system.

Why does conversational rewriting matter?

Follow-up questions in chat are fragments: "what about the other one," "does that apply here." Rewriting them into standalone queries using conversation history gives retrieval the full intent. Without it, retrieval searches for fragments and returns noise. Conversation design is in how to build an ai chatbot.

How do hypothetical document embeddings work?

The model drafts a plausible answer to the question, that draft is embedded, and retrieval finds passages similar to the hypothetical answer rather than to the question. Because answers resemble documents more than questions do, this can improve semantic retrieval, at the risk of embedding a hallucinated answer. It suits corpora where questions and content differ in form. Vector retrieval is in what is a vector database.

What does query rewriting cost?

An extra model call before retrieval adds latency and tokens, multi-query generation multiplies retrieval calls, and poorly constrained rewriting can drift from intent. Small, fast models or rules handle common rewrites; larger models handle decomposition. Apply techniques where evaluation shows gains. Latency budgeting is in what is latency in ai systems and cost control in llm api cost optimization.

How should query rewriting be evaluated?

With a golden set of raw user queries and relevant passages, comparing retrieval recall and precision with and without each rewriting technique, inspecting rewritten queries for drift, and measuring end-to-end answer quality. Log every rewrite so failures can be traced. Evaluation practice is in what is a golden dataset and reranking downstream in what is a cross-encoder.

What does query rewriting look like in practice?

A benefits assistant received terse queries such as "dental kids" and follow-ups such as "and for my spouse." Clarification and expansion turned the first into a query with plan terminology; conversational rewriting turned the second into a standalone question; filter extraction pulled plan year into metadata. Recall on the golden set rose and wrong answers from fragment queries fell, with a small fast model keeping added latency low. Chunk design that pairs with rewriting is in what is chunking in rag.

When should query rewriting be skipped?

When queries are already precise, such as identifiers or exact titles, when latency budgets are tight and retrieval quality is adequate, or when rewriting introduces drift on a query class. Evaluate the contribution per query type and disable rewriting where it does not help.

How FISTA Solutions applies query rewriting

FISTA Solutions adds rewriting techniques selectively where client golden sets show retrieval misses, uses fast models or rules for common rewrites, logs every rewritten query for evaluation, and measures end-to-end groundedness rather than retrieval metrics alone. The AI enablement practice delivers retrieval platforms, AI agents draw on them, and forward deployed engineers embed with client knowledge teams. The record behind the approach is 150+ projects with 99.9% uptime.

To make retrieval understand what users actually ask, message FISTA on WhatsApp, or read what is graph rag for retrieval over relationships rather than passages.