CI from scratch on Nx affected: from “no pipeline” to 9 minutes for all staging apps

Point zero: deploying by hand

At first there was no pipeline. There was a shell script: build the app locally and copy the files to the server. The problems are obvious:

• what gets built is what’s on your disk, not what’s in the repository;
• no checks before deploying;
• only someone with access and the script can deploy.

The first pipeline repeated the script: install dependencies → build all apps → copy. Better, but all apps were built on every push.

The idea of affected, in plain words

Nx knows the project graph: which app depends on which libraries. If you change a file in the users library, Nx walks the graph “upwards” and finds every app that uses that library. The rest aren’t affected; they don’t need to be built or tested.

npx nx affected -t build --base=<коммит> --head=HEAD

Nx takes the files changed between base and head (essentially git diff), finds their projects, adds everything that depends on them, and runs build only for that set.

It’s an ordinary graph traversal. If libraries are vertices and imports are edges, then “affected” means every vertex from which you can reach the changed one along the edges.

The main question: what is base?

The command is simple, but it’s easy to get base wrong. The answer differs from task to task.

For checking an MR: the branch point

For a merge request, the base is the commit where the branch split off from the target (the merge-base). In GitLab that’s the CI_MERGE_REQUEST_DIFF_BASE_SHA variable.

An important detail: by default GitLab clones the repository with a limited history depth. Without the full history git can’t find the merge-base, and affected computes the wrong set. So:

variables:
  GIT_DEPTH: "0"   # полная история

For the staging build: the last deployed commit

The first thought is to compare with the branch’s previous commit (CI_COMMIT_BEFORE_SHA). That’s a mistake, and here’s why.

Our staging pipelines are interruptible (interruptible: true): if a new push arrives, the old build is cancelled — the new one will replace it anyway. Imagine:

1. Commit A changes app X. Pipeline A starts.
2. A minute later commit B changes app Y. Pipeline A is cancelled, B starts.
3. B compares itself with A and builds only Y.

The changes to X never made it into any build. The server has the old version of X, and nobody knows why.

The right base is the last commit that actually reached the server. A separate job records it after a successful deploy:

# учебный пример
mark-deployed:
  stage: mark_deployed
  needs: [deploy]
  cache:
    key: deployed-sha
    paths: [.ci-cache/]
    policy: push
  script:
    - mkdir -p .ci-cache
    - echo "$CI_COMMIT_SHA" > .ci-cache/deployed-sha

The build reads this file and computes affected from it. It’s written only after a successful deploy, so changes from failed and cancelled pipelines automatically go into the next build.

And always — a fallback: build everything, if:

• there’s no base (first run, cache cleared);
• that commit isn’t in the history (for example, after a force-push);
• the CI itself or the list of apps changed;
• someone started the pipeline with FULL_BUILD=true.

Better to sometimes build too much than to once not build what’s needed.

Checking an MR: only what changed

Building is half of it. The other half is keeping errors out of staging. A verify stage on every MR, all by affected:

The three jobs run in parallel. Old linter violations are recorded in a baseline file and don’t get in the way; only new ones fail.

And the same thing locally with one command (pnpm verify). If a check exists only in CI, a person learns about the error ten minutes later. If it also exists locally — before the push.

Pitfall 1. Two builds in parallel — OOM

I wanted to speed things up: build two apps at a time (--parallel=2). The builds started failing with a strange esbuild error: all goroutines are asleep - deadlock!.

The error is misleading. The real cause is memory. Two parallel Angular builds took about 13 GB on a host with 15.5 GB. The Linux kernel (the OOM killer) killed the build process, and the esbuild child process, left without a parent, failed with that message.

Lesson: when CI shows a strange error, look at resources first — memory, disk, time. The fix is --parallel=1. Slower by one build, but affected had already reduced their number.

Pitfall 2. The cache that hid tests

Nx can cache results: if a task’s inputs haven’t changed, it doesn’t run it but restores the result from the cache (including the remote one, Nx Cloud).

Suddenly the MR report showed 237 tests out of 944. Nobody had deleted any tests. The reason: only a task’s declared outputs (outputs) are restored from the cache. The junit.xml file for the report wasn’t among them. A project from the cache is “green”, but without a report, and GitLab simply doesn’t show it.

The fix for tests in MRs is --skip-nx-cache. A report needs an honest run. The alternative is to add junit.xml to the task’s outputs.

Lesson: the cache speeds things up, but everything you later read from files must be in outputs. Otherwise it quietly disappears.

Pitfall 3. Building what nobody uses

Of the 17 apps in staging, only 7 were actually tested. The rest were built simply because they were on the list. We switched off the extra ones in the deploy map — more than half the work gone without a single line of clever code.

The fastest build is the one that doesn’t happen.

Pitfall 4. Editing CI in the web editor

I edited the first version of the pipeline right in GitLab’s web editor: in one week the history collected about twenty-five commits titled “Update .gitlab-ci.yml file”. Each one: run, wait, look, fix.

What to do differently:

• write CI in a branch and check it in an MR;
• validate the YAML with GitLab’s built-in linter (CI Lint) before pushing;
• move logic out of YAML into scripts (node tools/scripts/...) that can be run and debugged locally.

Small things that also save minutes

• A pnpm store cache keyed on pnpm-lock.yaml: dependencies aren’t downloaded from scratch every time.
• One job writes the cache, the others only read it (policy: pull): otherwise three parallel jobs take turns archiving and uploading the same thing.
• Source maps are uploaded to the error-monitoring system in the same job as the build and deleted before the deploy: the server doesn’t need them, and the artifact is smaller.

Summary

The main thing I took away: nx affected isn’t about the command but about the right answer to the question “compared to what?”. And about the “build everything” fallback when there’s no answer.

Related posts: nx-monorepo-from-copies, tailwind-v3-to-v4-monorepo, custom-eslint-rules.