What Is a Cross-Encoder? Precise Reranking for Retrieval

Embedding search is fast because documents are encoded once and compared by similarity, and it is imprecise for the same reason: the query and the document never meet inside the model. A cross-encoder puts them together, attending across both texts to judge relevance directly, which is far more accurate and far more expensive. The resolution is to let fast retrieval find candidates and the cross-encoder rerank them. This explainer covers how cross-encoders work and where they fit, drawing on FISTA Solutions' AI enablement practice. The pipeline role is in what is a reranker and the retrieval design in what is hybrid search.

What is a cross-encoder?

A cross-encoder is a transformer model that receives a query and a document concatenated as a single input and outputs a relevance score. Every token of the query can attend to every token of the document, so the model can weigh specific matches, negations, and context that separate encodings lose. It is trained on labeled query-document pairs to predict relevance. The architecture it builds on is described in what is a transformer model.

How does it compare with a bi-encoder?

Dimension	Bi-encoder (embedding model)	Cross-encoder
Inputs	Query and document encoded separately	Query and document encoded together
Output	Vectors compared by similarity	Direct relevance score
Precomputation	Documents embedded and indexed once	None; runs per pair at query time
Speed over a corpus	Fast, via vector index	Impractical
Accuracy	Good for meaning, weak on fine distinctions	High
Role	First-stage retrieval	Reranking a candidate list

Embedding retrieval is described in what is semantic search.

Why is joint attention more accurate?

Separate embeddings compress each text into a fixed vector before comparison, discarding details that matter only relative to a specific query. Joint attention keeps the details: whether the document addresses the exact entity, whether a negation flips the meaning, whether a number matches. This precision is why reranking with a cross-encoder often produces the largest single improvement in a retrieval pipeline.

Where does it fit in the pipeline?

First-stage retrieval, keyword, vector, or hybrid, returns a candidate pool of a few dozen to a few hundred passages; the cross-encoder scores each against the query; the top few proceed to the interface or the language model's context. Candidate pool size balances recall against reranking cost. Fusion before reranking is in what is hybrid search and lexical retrieval in what is bm25.

What does reranking cost?

One inference per pair, so latency and cost scale with candidates times passage length and model size. Small rerankers handle dozens of candidates within tens of milliseconds on suitable hardware; larger models are slower and more accurate. Hosted reranking services price per query or per document. Latency budgeting is in what is latency in ai systems.

How does a cross-encoder improve RAG?

By putting the genuinely relevant passages at the top, reranking lets a RAG system send fewer, better passages to the model, which improves groundedness, reduces tokens and cost, and cuts wrong answers from plausible-but-irrelevant context. Groundedness effects are in what is groundedness in ai and retrieval improvement in how to improve rag accuracy.

When should a cross-encoder be fine-tuned?

When the domain's notion of relevance differs from general web relevance, such as legal, medical, or technical corpora, and when labeled query-passage pairs can be collected from search logs, golden sets, or expert annotation. Fine-tuned rerankers can outperform much larger general models on domain queries. Labeling practice is in what is a golden dataset.

How do language models compare as rerankers?

Prompting a language model to score or order candidates is cross-encoding with more flexibility: complex relevance criteria, multi-aspect judgments, explanations. It costs more per pair and adds latency, so dedicated cross-encoders handle routine reranking and language models handle special criteria or final selection. Judge patterns are in what is llm-as-a-judge.

What does cross-encoder reranking look like in practice?

A policy assistant retrieves fifty hybrid candidates, reranks them with a cross-encoder, and passes the top five to the model. Compared with sending the top ten unreranked vector results, groundedness scores rise, tokens per answer fall, and wrong answers from near-miss passages drop. A fine-tuned reranker on the organization's own query logs improves results further. System design is in how to build a rag system.

How FISTA Solutions uses cross-encoders

FISTA Solutions adds cross-encoder reranking to retrieval pipelines by default, sizes candidate pools against latency budgets, fine-tunes rerankers where domain relevance and labeled data justify it, and measures gains on client golden sets and end-to-end groundedness. The AI enablement practice delivers retrieval platforms, AI agents draw on reranked context, and forward deployed engineers embed with client search and knowledge teams. The record behind the approach is 150+ projects with 99.9% uptime.

To get the right passages to the top, message FISTA on WhatsApp, or read what is query rewriting for the step that improves what the reranker has to choose from.