How to Build a CI/CD Pipeline for Machine Learning (Playbook)

Software CI/CD asks whether the code compiles and the tests pass. Machine-learning CI/CD asks a harder question: does the system still behave correctly on the cases that matter, given this code, this data, these features, this prompt, and this base model? A CI/CD pipeline for machine learning answers it on every change and ships safely. This playbook covers the build for classical models and LLM applications, following FISTA's AI enablement practice. Foundations are in what is mlops, devops vs mlops, and llmops vs mlops.

What does the pipeline do?

Stage	Function
Versioning	Code, data snapshots, features, prompts, configs, base models
Data validation	Schema, distribution, freshness, leakage checks
Build and test	Unit and integration tests on pipeline code
Training or configuration	Reproducible model training or LLM configuration assembly
Evaluation gate	Golden dataset, safety suite, comparison with production
Registration	Candidate version with lineage and results
Approval	Automated or human by risk tier
Deployment	Shadow, canary, blue-green, or A/B
Monitoring and rollback	Behavioral monitoring; instant revert

Step 1: Version everything that affects behavior

Store code in version control; snapshot training and evaluation data with identifiers; version features through the feature store; version prompts and configurations through prompt management; pin base-model versions. Every pipeline run records the full set. See how to build a prompt management system and how to build a feature store.

Step 2: Validate data

Before training or evaluation, validate schemas, distributions against baselines, freshness, null rates, and leakage indicators. Upstream breaks are the most common cause of bad models, and they are cheapest to catch here. See ai training data checklist.

Step 3: Test code

Run unit tests on transformations and pipeline logic, integration tests on end-to-end pipeline execution with fixtures, and contract tests on serving interfaces. Treat pipeline code like production code. Practice is in ai quality assurance.

Step 4: Train or assemble reproducibly

Execute training or LLM configuration assembly in a reproducible environment with pinned dependencies and recorded parameters, producing an artifact or configuration with lineage. Reproducibility is what makes rollback and investigation possible.

Step 5: Gate on evaluation

Run the golden dataset and safety suite against the candidate, compare with the current production version by category, and fail the pipeline on threshold breaches. Handle non-determinism with repeated runs and pass-rate gates. Evaluation is the release gate that unit tests cannot be. Method is in the AI evaluation and testing whitepaper and how to build an agent evaluation harness.

Step 6: Register and approve

Register passing candidates in the model registry with lineage and results; apply approval by risk tier, automated for low-risk changes within thresholds and human for higher tiers. Deployment pulls only approved versions. See how to build a model registry.

Step 7: Deploy progressively

Choose the strategy by risk: shadow deployment to compare behavior on live traffic without effect, canary to a small fraction with monitoring, blue-green for instant switchover and rollback, and A/B tests for business-metric validation. Monitor behavioral metrics, not just health, at each stage. Concepts are in what is a canary deployment and what is a shadow deployment.

Step 8: Monitor and roll back

Connect production monitoring to versions; define automatic rollback triggers for error rates, latency, and quality drops; make rollback a pointer change to the previous approved version; and test rollback regularly. See the AI observability whitepaper.

Worked example: a support triage classifier and its LLM companion

A team operates a trained ticket classifier and an LLM-based response drafter. The pipeline versions the classifier's training data snapshots and code, and the drafter's prompts, retrieval configuration, and pinned base model. A retraining trigger runs data validation, which catches a schema change in the ticketing export before training; the corrected run produces a candidate that passes the per-category evaluation gate and is auto-approved as low risk, deployed as a canary, and promoted after monitoring holds. Separately, a provider announces a base-model update; the pipeline assembles a candidate configuration, runs the golden set and safety suite, finds a regression on escalation cases, and blocks promotion until the prompt is adjusted. Both systems roll back in seconds when needed because rollback is a registry pointer change exercised in a monthly drill.

What does it cost?

The pipeline costs build effort and evaluation compute per run, controlled by sampling on minor changes. It repays itself by preventing regressions, enabling frequent safe changes, and making incidents recoverable. See the AI total cost of ownership whitepaper.

What are the common mistakes?

• Versioning code but not data, prompts, or base models.
• No data validation, so upstream breaks become bad models.
• Unit tests as the only gate.
• Manual deployments that bypass the registry.
• Full-traffic launches without shadow or canary.
• Rollback that has never been tested.

How do you phase the pipeline?

Start by putting the current production model or LLM configuration through the pipeline as it exists, with versioning, data validation, and the evaluation gate, without changing the model itself; this exposes what is not reproducible and what the golden set is missing. Add registry-based promotion and progressive deployment next, then automate retraining triggers and drift-driven runs. Teams that begin with automated retraining before the gates exist automate the production of unverified models. The first successful rollback drill is the milestone that shows the pipeline is trustworthy.

Who owns the pipeline?

The platform team owns the pipeline framework and gates; model and application teams own their evaluation suites and thresholds; governance owns approval requirements by risk tier. Shared ownership without these boundaries produces gates nobody maintains.

How do you keep the pipeline trustworthy over time?

Treat the pipeline itself as a product: version its configuration, alert when evaluation datasets or thresholds change, audit who approved each promotion, and review gate failures monthly to distinguish real regressions from flaky checks. Pipelines that nobody maintains drift into rubber stamps, and the first sign is usually a promotion that nobody can explain.

How FISTA Solutions builds ML pipelines

FISTA Solutions builds CI/CD pipelines for machine learning and LLM applications to this playbook as part of its AI enablement platform: complete versioning, data validation, tested pipeline code, reproducible builds, evaluation gates with non-determinism handling, registry-based promotion, progressive deployment, and tested rollback. Every AI agent FISTA delivers ships through such a pipeline, and forward deployed engineers integrate it with your existing delivery tooling. The record behind the work is 150+ projects with 99.9% uptime.

To scope an ML CI/CD pipeline, message FISTA on WhatsApp, or read ai devops for the wider operating context.