Data Readiness for Generative AI: A Whitepaper

Every generative AI system is a function of the content it can reach. A retrieval system over poorly parsed documents with no metadata and no permission mapping will produce confident, unauthorized, out-of-date answers, and no model choice will fix it. This whitepaper defines data readiness for generative AI, distinguishes it from traditional data readiness, and gives a method for assessing and closing gaps before they become production failures. The general concept is introduced in AI data readiness; this whitepaper goes deeper on the generative case.

How does generative AI change the data readiness question?

Traditional AI readiness concerned structured data: tables, schemas, pipelines, feature quality. Generative AI adds three new classes of input:

1. Unstructured content: documents, presentations, wikis, tickets, emails, contracts, transcripts, with structure that must be preserved in parsing.
2. Conversational and operational context: CRM notes, support threads, chat history, with provenance and freshness questions.
3. Tacit knowledge: the rules and exceptions that live in experienced employees' heads and were never written down.

Each requires different readiness work, and the third requires discovery rather than engineering. The retrieval architecture these feed is described in the enterprise RAG reference architecture whitepaper.

What are the six readiness dimensions?

Dimension	Question	Typical gaps
Access	Can we reach the content programmatically, incrementally, and with change detection?	No APIs; export-only; no deletion events
Structure	Does parsing preserve headings, tables, lists, and relationships?	Flattened PDFs; scanned images; slide decks
Metadata	Do chunks carry source, date, author, type, unit, version, and sensitivity?	Missing dates; no ownership; no versioning
Permissions	Are access rights known per document and enforceable at retrieval?	Inconsistent ACLs; shared drives with broad access; orphaned owners
Quality and freshness	Is content accurate, current, deduplicated, and free of contradictions?	Stale policies; duplicates; conflicting versions
Governance	Are sensitivity, retention, residency, and usage rules defined and applied?	No classification; unclear retention; provider-terms conflicts

How is access readiness assessed?

For each source, establish whether content can be retrieved through a supported connector or API, whether incremental updates and deletions are detectable, what authentication is required, and what rate limits or export constraints apply. Sources that only support manual export are readiness gaps: they will go stale. Access readiness also covers the organizational dimension: who owns the source and can authorize connection.

What does structural readiness require?

Parsing must preserve meaning. Common failures: tables flattened into word soup, headings lost so chunks have no context, scanned documents with no text layer, slide decks parsed into disconnected fragments, and embedded images carrying key information. Structural readiness work includes selecting or building parsers per format, adding optical character recognition where needed, capturing document hierarchy, and testing parsing quality on representative samples. Guidance is in document processing AI and how to build an OCR pipeline with LLMs.

Why does metadata matter so much?

Metadata powers filtering (restrict to a business unit or date range), freshness (prefer or require current versions), citation (show the user where the answer came from), permissions (see below), and evaluation (stratify results by source type). Content without provenance cannot be cited, and content without dates cannot be trusted. Metadata readiness work includes extracting what exists, inferring what can be inferred, and establishing ownership for what is missing.

How is permission readiness assessed and closed?

Permissions are the most neglected dimension and the most dangerous to skip. Assessment questions:

• Does each source expose access-control information per document?
• Is the permission model consistent with the identity provider's groups and roles?
• Are there sources with broad or inherited access that do not reflect intended policy?
• Can entitlements be resolved at query time for a given user?
• How are permission changes propagated?

Closing gaps may require cleaning up the source permissions themselves, which is organizational work, before the AI system can safely retrieve from them. The enforcement architecture is described in AI access control.

How are quality and freshness assessed?

Quality assessment samples content for accuracy, currency, duplication, and contradiction. Typical findings: multiple versions of the same policy with different rules, outdated procedures still marked current, duplicated documents across drives, and content that contradicts the system of record. Remediation options include designating authoritative sources, archiving superseded content, deduplicating, and adding version metadata. Freshness readiness also requires a refresh strategy per source, aligned with how fast the content changes.

What governance readiness is required?

Governance element	Requirement
Classification	Every source classified by sensitivity; handling rules per class
Retention	Retention rules for source content, prompts, outputs, and traces
Residency	Processing and storage locations permitted for each class
Provider terms	Model-provider agreements consistent with the data's regulation; redaction where required
Usage policy	Which content may be used for which AI purposes
Lineage	Ability to trace outputs back to sources

Guidance is in AI data governance, AI data residency, and AI data privacy compliance.

How is tacit knowledge made ready?

Some of the most valuable knowledge, the rules by which experienced staff handle exceptions, exists nowhere in writing. Readiness for this class is a discovery activity: shadowing, case walkthroughs, and structured interviews that produce written rules, decision tables, and worked examples. These become both content for retrieval and the basis of the system's specification and evaluation dataset. This is a core reason forward deployed engineers embed with operators; the method is in the spec-driven development for AI whitepaper.

How should a readiness assessment be run?

1. Scope to the first use case. Identify the sources it actually needs; do not attempt an enterprise-wide inventory first.
2. Score each source on the six dimensions with evidence from samples and system inspection.
3. Identify tacit knowledge required and plan discovery.
4. Estimate remediation per gap: engineering (connectors, parsers), data work (metadata, deduplication), organizational (permission cleanup, ownership), governance (classification, terms).
5. Sequence remediation so the first workflow can proceed while longer items continue.
6. Report readiness with a remediation plan and its cost, so the business case reflects reality.

A condensed instrument is the AI data readiness checklist, and the broader organizational assessment is AI readiness assessment.

How does readiness connect to cost and success?

Data readiness work is routinely the largest build-phase cost in generative AI projects and the strongest predictor of whether they reach production. Business cases that omit it overrun; programs that assess it honestly sequence correctly. The cost structure is examined in the AI total cost of ownership whitepaper, and the sequencing in the enterprise AI adoption roadmap whitepaper.

What are the common mistakes?

• Indexing everything on a shared drive without permission mapping.
• Treating parsing as solved because text came out of the PDF.
• Ignoring deletions and updates, so the index diverges from reality.
• Assuming the wiki is current.
• Sending sensitive content to a model provider under terms that do not permit it.
• Skipping discovery of tacit rules and discovering them as production errors.
• Trying to make all data ready before delivering any use case.

Worked example: readiness assessment for an internal policy assistant

Consider an organization that wants an assistant to answer employee questions about HR, finance, and IT policies. The use case seems simple, and the readiness assessment shows why it is not. Access: policies live in a document management system with an API, a wiki with export only, and a shared drive with neither; the wiki and drive are gaps. Structure: policy PDFs parse cleanly, but the benefits tables flatten and lose meaning; a table-aware parser is required. Metadata: effective dates exist for some policies and not others, and ownership is unrecorded for a third of documents; owners must be assigned before freshness can be governed. Permissions: most policies are company-wide, but manager-only guidance and regional variations sit in the same folders with inherited broad access; permission cleanup in the source is required before indexing. Quality and freshness: three versions of the expense policy coexist, two superseded; an authoritative-source designation and archiving are needed. Governance: the policies are internal but the assistant will receive employee questions that may include personal data, so prompt handling and provider terms must be reviewed. Tacit knowledge: the HR team applies unwritten exceptions for certain cases; these are documented through discovery and become both content and evaluation cases. The readiness report sequences the remediation so the assistant can launch on the document management system's policies first, with the wiki and drive following once connectors and permissions are fixed, and the business case reflects the real work rather than the hoped-for shortcut.

What does readiness look like a year in?

Sources have named owners and documented semantics, refresh pipelines run with quality checks and alerts, permission metadata travels with content, golden sets exist for each use case, and new sources are onboarded through a repeatable process rather than a project. Readiness is a capability the organization keeps rather than a state it reaches once.

How FISTA Solutions approaches data readiness

FISTA Solutions runs data readiness assessment as the first step of every generative AI engagement, scoped to the first workflow, scored across the six dimensions, and reported with a remediation plan and cost. Forward deployed engineers surface tacit knowledge with your operators and turn it into specifications and evaluation sets; the AI enablement practice builds the connectors, parsing, metadata, and permission-aware retrieval that make content usable; and AI agents are deployed only on sources that meet the readiness bar. The approach is backed by 150+ projects delivered with 99.9% uptime.

To assess data readiness for a specific generative AI use case, message FISTA on WhatsApp, or read how to prepare data for AI for a practical starting point.