CI/CD Pipeline

Problem: pip install ran from scratch on every CI run. For the eval service with its 200+ transitive dependencies, this took ~20 minutes — longer than all other checks combined.

Investigation:The GitHub Actions runner starts fresh each time, so there was no pip cache to reuse. The eval service's former third-party evaluation dependency tree made cold installs exceptionally slow.

Fix: Cache the entire .venv directory using actions/cache@v4, keyed on the hash of requirements.txt and shared/pyproject.toml. On cache hit, the install step is skipped entirely.

# Cache key
venv-{service}-{hash(requirements.txt, shared/pyproject.toml)}

# On cache hit → skip pip install entirely

Result: Eval tests went from 20 minutes → 20 seconds. pip-audit dropped from 20 minutes → 9 seconds.

Problem: All 11 service images were rebuilt on every push, even when only one service changed. A one-line fix to the chat service triggered builds for all Go, Java, and Python images.

Investigation: The build matrix had no path awareness — every matrix entry ran unconditionally. Most builds were wasted compute producing identical images.

Fix: Each matrix entry declares a paths field listing the directories that affect its image. A git diff HEAD~1 check at the start of each build job skips the build when none of those paths changed.

- service: chat
  paths: services/chat services/shared
- service: go-auth-service
  paths: go/auth-service go/pkg go/go.work

Result: A typical single-service change rebuilds 1 image instead of 11. Unchanged services are skipped in ~20 seconds.

Problem: docker compose up --build rebuilt all Python images from source in CI, spending ~10 minutes per service on pip install with no layer cache (fresh runner each time).

Investigation:The compose-smoke job existed to verify service configuration — env vars, nginx routing, health checks, inter-service connectivity. It didn't need freshly built images to test those things. Code correctness was already covered by unit tests.

Fix: Pull pre-built :latest images from GHCR instead of building from source. The smoke tests verify configuration, not code.

# Before: build from source (~15 min)
docker compose up --build

# After: pull pre-built images (~95 sec)
for svc in ingestion chat debug; do
  docker pull "ghcr.io/.../${svc}:latest"
done
docker compose up -d

Result: Compose smoke went from ~15 minutes → 95 seconds.

Problem: QA deploys were failing entirely. The Go kustomize overlay includes migration Jobs, and Kubernetes Jobs are immutable — once created, their spec.template cannot be patched.

Investigation: When kustomize apply tried to update existing Jobs with a new image tag, Kubernetes rejected it with field is immutable. The Jobs had completed successfully on the previous deploy but were still present in the namespace.

Fix: Filter Jobs out of the kustomize output using awk, then handle them separately: delete the old Job, create the new one, wait for completion.

# Apply overlay without Jobs
kubectl kustomize k8s/overlays/qa-go/ \
  | awk '...filter out kind: Job...' \
  | kubectl apply -f -

# Run migrations sequentially
kubectl delete job go-auth-migrate --ignore-not-found
kubectl apply -f auth-service-migrate.yml
kubectl wait --for=condition=complete job/go-auth-migrate

Result: QA deploy went from failing → succeeding in 85 seconds.

Problem: The path filter from #2 evolved with the codebase. The first version compared against HEAD~1, which silently missed rebuilds when a fix was pushed in a multi-commit batch (the diff only saw the final commit). A post-incident hardening widened it to HEAD~5. That fixed the missed-rebuild bug but introduced the opposite failure mode: once Go work landed in the last 5 commits, every subsequent push — including docs-only or frontend-only ones — re-ran the full Go test, lint, and image-build matrices.

Investigation: GitHub gives the exact range for every event: pushes carry github.event.before (the previous tip of the branch); PRs carry github.event.pull_request.base.sha (the merge base). Both are precise — HEAD~N was always a heuristic dressed up as a window. The same overshoot also hit the test and lint matrices, which never had path filtering at all and re-ran the full Python, Java, and Go test suites on every push regardless of what changed.

Fix: A composite action at .github/actions/check-changes picks the compare base based on the event type — push.before for pushes, PR base SHA for PRs, with HEAD~5 kept only as a fallback for first pushes and workflow_dispatch. Wired into 14 gated jobs: the original three (go-tests, go-lint, build-and-push-images) plus python-tests, java-unit-tests, java-integration-tests, frontend-checks, k8s-manifest-validation, go-migration-test, all three compose-smoke jobs, security-pip-audit, and security-hadolint. Every gated entry's paths: value includes ci.yml and the action's own action.yml, so a workflow refactor triggers every matrix entry — a safeguard against silent pipeline regressions.

- name: Check for changes
  id: changes
  uses: ./.github/actions/check-changes
  with:
    paths: services/chat services/shared
           .github/workflows/ci.yml
           .github/actions/check-changes/action.yml

# Subsequent steps gated on:
#   if: steps.changes.outputs.changed == 'true'

Result: A docs-only push now skips ~14 matrix entries in seconds. The pipeline failure mode shifted with each iteration: HEAD~1 silently missed rebuilds, HEAD~5 silently over-rebuilt, and the precise push-range / PR-base approach rebuilds exactly what changed.

Lesson: The first merge after extending the gate broke CI entirely. Every run showed “This run likely failed because of a workflow file issue” with an empty workflow graph — no jobs visible because nothing ever started. Locally, the pre-flight check used python -c "yaml.safe_load(...)", which passed because YAML accepts duplicate keys (last value wins). actionlint caught it instantly: a step had both the new gate if: steps.changes.outputs.changed == 'true' and an existing if: steps.venv-cache.outputs.cache-hit != 'true' — two if:keys on the same map, which GitHub's parser rejects. Validate workflow files with actionlint, not just a YAML parser. YAML's permissiveness is the wrong shape for CI configs.