Playbook · 5 minute read
How to Build an Internal Developer Agent
An internal developer agent answers engineers' questions about the organization's codebases, architecture, and processes by retrieving from repositories, documentation, runbooks, and decision records, citing its sources, and optionally taking bounded actions such as opening issues or drafting changes under the same gates as any coding agent. It is built retrieval-first, evaluated on real questions, and permissioned narrowly.
Engineers spend a surprising share of their week asking questions the organization already knows the answer to: how a service handles a case, where a configuration lives, why a design decision was made, how a deployment is run. The answers are scattered across repositories, documents, decision records, and the heads of senior engineers who are interrupted to provide them. An internal developer agent answers those questions from your sources, with citations, and later takes bounded actions. This playbook builds one. It applies the enterprise RAG reference architecture to engineering knowledge and the governance of the agentic SDLC whitepaper to any actions.
Step 1: Which questions should it answer first?
Collect a month of questions from onboarding channels, support rotations, and senior engineers' direct messages. Cluster them. The first release targets the most frequent, answerable-from-sources clusters: service overviews, configuration locations, ownership, how-to-run procedures, and the rationale behind decisions. Questions requiring judgment about future design stay with people.
Step 2: What should be indexed?
| Source | Value | Notes |
|---|---|---|
| Repositories: code, READMEs, inline docs | Ground truth for how things work | Chunk by file and symbol; keep paths |
| Architecture docs and diagrams | Structure and intent | Version them; mark stale ones |
| Decision records | Why | Highest-value source for "why" questions |
| Runbooks and playbooks | How to operate | Critical for on-call questions |
| Issue and pull request history | Context and precedent | Index summaries; link to originals |
| Ownership metadata | Who | From code-owner files and team registries |
Exclude secrets, credentials, and personal data from indexing entirely; scanning before indexing is mandatory.
Step 3: How do permissions work?
The agent retrieves under the asking engineer's identity, so it searches only repositories and documents that engineer may read. Restricted repositories are opted in by their owners. No answer is cached across permission boundaries. The delegated-identity pattern is described in the agent identity and access control whitepaper.
Step 4: Build the retrieval layer
Use hybrid retrieval, lexical and semantic, over code-aware chunks, with metadata filters for repository, path, language, and recency, and a reranker for precision. Code and prose need different chunking: symbols and files for code, sections for documents. Retrieval quality is scored separately from answer quality so failures are diagnosable. The pattern is described in how to build a hybrid search system.
Step 5: Generate grounded answers
Every answer cites the files and documents it drew on, with paths and links, and states when sources conflict or are stale. Ungrounded answers are prohibited: if retrieval finds nothing relevant, the agent says so and suggests who to ask from ownership metadata. Answers that touch operational procedures quote the runbook rather than paraphrasing it.
Step 6: How do you evaluate it?
Build a golden set from the collected questions with verified answers and the sources that support them. Score retrieval (were the right sources found), groundedness (does the answer follow from the sources), and correctness (is it right), by question cluster. Gate every change to the index, retrieval, prompts, or model on regression against the set. Sample production answers weekly for engineer review and feed corrections back. The method is evaluation-driven development.
Step 7: Add bounded actions later
Once question answering is trusted, add actions under the coding-agent policy: open an issue with gathered context, draft a documentation fix as a pull request, propose a runbook update. Actions run under scoped identities, pass the standard gates, and are reviewed like any generated change; the policy is in AI coding agent governance policy. Never let the agent merge or deploy.
How should the agent fit into engineers' workflow?
An agent nobody reaches for answers nothing. Put it where questions already happen.
| Surface | Use | Notes |
|---|---|---|
| Chat channel | Onboarding and support questions in the open, so answers are visible to others | Threaded replies with citations; a reaction to rate the answer |
| IDE or editor | "Explain this module" and "where is this configured" from the code itself | Same retrieval and permissions; keep it read-only at first |
| Command line | Scripted lookups during on-call: runbook steps, ownership, recent changes | Ideal for operations questions |
| Documentation site | Search box that answers rather than lists | Every answer links to the page it came from |
Two conventions matter. The agent should answer in the channel where the question was asked rather than pulling engineers into a separate tool, and every answer should carry a one-click way to say it was wrong, because those signals are the cheapest source of evaluation data you will ever get. Route corrections to the source owner as documentation requests, not only to the agent's maintainers; the agent's most common failure is a stale or missing document, and the fix belongs upstream.
Step 8: Roll out and measure
Launch in the onboarding channel and one team's support rotation. Measure time to first contribution for new engineers, questions answered without a senior engineer, citation click-through, answer ratings, and the escalation rate to humans. Track the index's freshness and the share of questions with no relevant sources, which is a documentation backlog signal.
Step 9: Operate
Re-index on repository changes, expire stale documents, review permission opt-ins quarterly, and keep the golden set growing from real questions. Treat gaps the agent reveals as documentation work for the teams that own the sources.
What are the common mistakes?
- Indexing everything, including secrets.
- Answers without citations.
- Ignoring permissions, so the agent becomes a way around repository access.
- Actions before trust.
- No golden set, so quality is anecdotal.
- Stale sources answered confidently.
How does FISTA Solutions help?
FISTA Solutions builds internal developer agents as governed AI agents on the retrieval and identity platform its AI enablement practice establishes, with forward deployed engineers working inside your engineering organization to collect questions, curate sources, and build the evaluation set. FISTA has delivered 150+ projects for 50+ companies across 12+ countries.
To build one for your engineering team, message FISTA on WhatsApp, or read how to build a Confluence knowledge agent for the documentation-side counterpart.
Share-ready article cover
Download the generated social format.
Clear answers
Questions raised by this field note.
Straightforward guidance for evaluating scope, fit, and the next step.
01What does an internal developer agent do?
It answers questions about your codebases and engineering processes from your own sources: how a service works, where configuration lives, why a decision was made, how to run a deployment, who owns a module. It cites the files it drew on and, once trusted, can take bounded actions such as opening an issue or drafting a change.
02How is it different from a coding assistant in the IDE?
An IDE assistant works on the code in front of the developer. An internal developer agent works across the organization's knowledge: many repositories, documentation, decision records, runbooks, and history, with permissions and citations. The two complement each other; the internal agent answers the questions the IDE assistant cannot see.
03How do you keep it from exposing code to the wrong people?
By enforcing repository permissions at retrieval time: the agent searches only what the asking engineer is authorized to read, using their identity, and never caches answers across permission boundaries. Secrets are excluded from indexing entirely, and restricted repositories are opted in explicitly.
04How do you know it is giving good answers?
Build a golden set of real questions with verified answers and sources, score retrieval and answer correctness separately, gate every change on it, and sample production answers for review. Engineer feedback on each answer feeds the set. Answers without citations are treated as failures regardless of content.
Continue exploring
Related capabilities
Start with the hard problem
Need the outcome owned, not merely analyzed?
Tell us where delivery is constrained. We’ll map the fastest credible path from intent to verified production.