CI/CD for LLM Applications with GitHub Actions

Shipping LLM applications without CI/CD discipline means every prompt edit is a production gamble. Here is a robust GitHub Actions pipeline for LLM apps — testing, evaluation, and deployment with rollback.

What Makes LLM CI/CD Different

Classic CI verifies functional correctness: does the code work? LLM CI must also verify behavioral correctness: does the system still answer well after this prompt tweak, model upgrade, or retrieval change? That means evaluation suites are part of the pipeline, not an offline activity.

The Pipeline Shape

name: deploy
on:
  push:
    branches: [main]

jobs:
  test:
    runs-on: ubuntu-latest
    steps:
      - uses: actions/checkout@v4
      - run: pip install -e ".[dev]"
      - run: pytest tests/ -x -q          # unit + contract tests
      - run: python -m evals.run --suite smoke   # 20-case fast eval

  evaluate:
    needs: test
    runs-on: ubuntu-latest
    steps:
      - uses: actions/checkout@v4
      - run: python -m evals.run --suite golden --report eval.json
      - run: python -m evals.gate eval.json --min-faithfulness 0.85 --max-regression 0.03

  deploy-canary:
    needs: evaluate
    environment: production
    steps:
      - run: ./deploy.sh --traffic 5
      - run: python -m evals.live --window 30m --auto-rollback

Layer 1: Contract Tests

Unit-test everything deterministic: parsers, chunkers, prompt templates render correctly, output schemas validate. Mock the LLM — these tests must be fast and free.

Layer 2: Golden-Dataset Evaluation

A few hundred curated cases with expected properties (not exact strings): must-cite sources, must-abstain cases, known-answer questions. Gate the deploy on absolute thresholds and regression vs. the current production baseline. Cache LLM-judge results by (case, prompt-version, model-version) to keep costs sane.

Layer 3: Canary with Live Evaluation

Route 5% of traffic to the new version, score sampled responses with the same judges, and auto-rollback if quality or cost regress. Behavioral changes that pass offline evals can still fail on real query distributions.

Version Everything Together

A deployable unit is: code + prompt templates + model IDs + index version + eval baseline. Tag them as one release. When someone asks "what changed Tuesday?", the answer must be one diff, not archaeology across four systems.

Secrets and Cost Hygiene

• Scope API keys per environment; never share prod keys with CI
• Set spend alerts on the CI project — a broken retry loop in evals can burn real money
• Fail the build if projected per-query cost jumps beyond a set percentage

Rollback is a Feature

Keep the previous release warm and make rollback one command. With LLM systems you will roll back for reasons tests can't catch — a model provider's silent update, a tone regression, a compliance complaint. Speed of recovery matters more than perfection of gates.

Automate testing, evaluation, and deployment this way, and prompt changes become boring — which is exactly what production changes should be.