What Is a Retrieval Index? The Structure Behind AI Search

The index is where retrieval quality is determined and where most operational problems in grounded AI systems originate. Teams tune embedding models and prompts while the index is missing a third of the corpus because a connector broke in March. This explainer covers what an index needs and how it fails. It complements what is metadata filtering in rag and what is retrieval augmentation, and reflects FISTA Solutions' approach in AI agents delivery.

What does an entry hold?

More than a vector. The chunk text itself, for display and citation. Source document identity and position, so an answer can point at where it came from. Section context, so the chunk is interpretable. Effective and expiry dates. Entitlement scope. Version.

Each of those supports something: filtering, citation, access control, or currency. An index holding only vectors and text cannot do any of it.

Field	Enables	Missing causes
Source and position	Citation	Unverifiable answers
Section context	Interpretation	Ambiguous chunks
Effective dates	Currency filtering	Superseded answers
Entitlement scope	Access control	Unauthorised disclosure
Version	Change tracking	Untraceable drift
Indexed timestamp	Freshness monitoring	Silent staleness

Why is freshness decisive?

Because a stale index produces confident answers from superseded content and users cannot detect it. A policy updated last month that has not been reindexed will be cited as current, with the same authority as anything else.

That is worse than having no answer. An unanswered question prompts the user to look elsewhere; a confidently wrong answer from an outdated document ends the enquiry.

What happens on deletion?

It must propagate, and frequently does not. Deleting a source document while its chunks remain indexed means the system keeps citing content that no longer exists — which is confusing at best, and where the document was removed for legal or compliance reasons, a genuine problem.

Deletion propagation should be tested explicitly, because it is the update path least exercised in development and most likely to be missing.

How do indexes decay?

Silently. A connector's credentials expire and one source stops updating. An extraction pipeline breaks for one file type. A permissions change removes access to a folder. In each case the index simply stops receiving part of the corpus, with no error and no symptom except answers that are quietly incomplete.

Coverage monitoring — comparing document counts and indexed timestamps per source against expectation — is what catches this. It is rarely implemented and it is the single most useful operational control on an index.

How should updates work?

Incrementally where possible, with full rebuild available. Incremental updates keep freshness high at low cost; the rebuild path handles the changes that touch everything.

Change detection matters here: polling every source repeatedly is expensive, and event-driven updates from the source systems are both cheaper and fresher where they are available.

What must be planned early?

The rebuild. Embedding model changes, chunking strategy changes, and metadata schema changes all require reprocessing the corpus, and a system with millions of documents and no rebuild pipeline is locked to decisions made in its first month.

Building that path while the corpus is small is straightforward. Retrofitting it later is a project. See what is an embedding model.

What should you do first?

Compare the number of documents in your source systems against the number represented in your index, per source. In most deployments past their first months the numbers differ, and the gap is usually a broken connector nobody noticed.

How should indexes be organised?

By access boundary first, then by content type. Splitting by entitlement scope makes filtering cheaper and reduces the consequences of a filtering bug, since content the user may not see is in a different index rather than one predicate away.

Splitting by content type helps where retrieval behaviour differs: short reference entries and long narrative documents benefit from different chunking and sometimes different embedding models, and mixing them in one index forces a compromise on both.

What about multi-tenancy?

Tenant isolation should be structural rather than a filter value. A shared index with a tenant predicate works until a query is constructed without it, and the consequence is one customer's content surfacing for another. Separate indexes per tenant, or a database that enforces isolation below the query layer, removes that failure mode rather than defending against it.

The cost is operational: many small indexes are more work than one large one, and for a product with many customers that overhead needs designing rather than absorbing.

What about cost?

Index storage and query cost scale with the corpus and with how richly it is represented, and both are ongoing rather than one-off. Re-embedding on a model change is the largest single expense, which is another reason to plan the rebuild path before the corpus grows.

How FISTA Solutions helps

FISTA Solutions builds retrieval indexes with full metadata for filtering, citation, and entitlement, monitors per-source coverage and freshness to catch silent decay, tests deletion propagation explicitly, and establishes rebuild pipelines before corpora grow large, through AI agents, AI enablement, and forward deployed engineers. The record behind the approach is 150+ projects for 50+ companies with 99.9% uptime.

To keep your retrieval index accurate as your content changes, message FISTA on WhatsApp, or read what is metadata filtering in rag.