pipeline.md (17411B)
1 .conductor.yml reference 2 ======================== 3 4 The pipeline file is read from the repository at the commit being built, so 5 it is versioned with the code it builds. The conductor never executes 6 anything from the repository while parsing it; the file is data. 7 8 Every unknown key is an error rather than a warning, and all problems in a 9 file are reported together. 10 11 Top level 12 --------- 13 14 | Key | Required | Description | 15 | ------------ | -------- | --------------------------------------------------- | 16 | `version` | yes | Must be `1`. | 17 | `workdir` | no | Absolute path tasks run in. Overrides the project. | 18 | `defaults` | no | Values inherited by every task. | 19 | `tasks` | yes | Mapping of task name to task. | 20 21 The file may be named `.conductor.yml` or `.conductor.yaml`; both are 22 looked for unless the project was pointed at some other path explicitly. 23 Carrying both at once is an error rather than a coin toss, since two 24 pipelines in one repository will disagree sooner or later. 25 26 This key was called `jobs` before 0.3.0. A job is now the pipeline run 27 that these tasks belong to, and a file still using the old spelling is 28 refused with a message saying so. 29 30 Task names may contain letters, digits, underscore, dot and hyphen, and must 31 start with a letter or digit. 32 33 ```yaml 34 version: 1 35 defaults: 36 image: debian:bookworm-slim 37 tasks: 38 test: 39 script: [make test] 40 ``` 41 42 Visibility 43 ---------- 44 45 Visibility is a project setting, not a pipeline one. It is recorded on each 46 job as it is created, so turning a project private stops exposing future jobs 47 without retroactively hiding, or revealing, earlier ones. 48 49 A `visibility` key here is rejected: a repository that could publish its own 50 results would let anyone able to land a commit read a private project's logs 51 and artifacts. Set it with `PATCH /api/v1/projects/:id` instead. 52 53 A public job, its logs and its artifacts are readable by anyone. A private 54 job is readable by the project's owner and by administrators. 55 56 Tasks 57 ---- 58 59 | Key | Type | Default | Inheritable | 60 | --------------- | ----------------- | -------------- | ----------- | 61 | `image` | string | none, required | yes | 62 | `script` | list of strings | none, required | no | 63 | `needs` | list | `[]` | no | 64 | `after` | list | `[]` | no | 65 | `arch` | string or list | none | yes | 66 | `matrix` | mapping | none | no | 67 | `requires` | list of strings | `[]` | yes | 68 | `services` | list | `[]` | yes | 69 | `env` | mapping | `{}` | merged | 70 | `artifacts` | list or mapping | none | no | 71 | `allow_failure` | boolean | `false` | yes | 72 | `timeout` | duration | server default | yes | 73 | `max_attempts` | integer, 1 to 10 | `1` | yes | 74 | `only` | mapping | none | no | 75 76 `image` and `script` are the only required fields, and `image` may come from 77 `defaults`. 78 79 ### script 80 81 A list of shell commands, run in order in the task's container. The task fails 82 on the first command that exits non-zero. 83 84 Take care with YAML scalars. Unquoted `true`, `false`, `yes`, `no` and `on` 85 parse as booleans, not strings, and are rejected: 86 87 ```yaml 88 script: ['true'] # correct 89 script: [true] # rejected, this is a boolean 90 ``` 91 92 ### timeout 93 94 Accepts whole seconds, or a duration with units `s`, `m`, `h`, `d` and `w`, 95 optionally combined: `90`, `90s`, `30m`, `1h`, `1h30m`. A task timeout may not 96 exceed seven days; `artifacts.expire` may not exceed a year. 97 98 Fanning out 99 ----------- 100 101 ### only 102 103 Restricts a task to certain refs. Without it a task runs on every push. 104 105 ```yaml 106 publish: 107 needs: [build] 108 only: 109 refs: [refs/heads/main, 'refs/tags/v*'] 110 script: 111 - ./deploy/publish.sh 112 ``` 113 114 Patterns match the whole ref, so `refs/heads/main` rather than `main`. 115 A `*` stands for any run of characters; everything else is literal, so a 116 dot is a dot rather than any character. 117 118 A task that does not match is left out of the job entirely, not recorded 119 as skipped. A skipped task still satisfies `needs`, so everything below it 120 runs anyway, which is the wrong reading for a publish step that was never 121 meant to run on this branch. 122 123 Anything depending on an excluded task is excluded with it. Dropping the 124 dependency instead would let a task run without something it declared it 125 needed, which is the worse surprise: 126 127 ```yaml 128 # On a branch, none of these three run. 129 publish: 130 only: { refs: [refs/heads/main] } 131 script: ['make publish'] 132 announce: 133 needs: [publish] 134 script: ['make announce'] 135 ``` 136 137 A job triggered without a ref matches no pattern, so restricted tasks stay 138 out rather than being handed a ref they were never written for. 139 140 The rule is not inheritable from `defaults`. A restriction that silently 141 applied to every task is hard to spot when the symptom is an empty job. 142 143 ### arch 144 145 Expands the task into one task per architecture, and restricts each to a worker 146 that offers it. The value is available to the script as `$ARCH`. 147 148 ```yaml 149 build: 150 arch: [x86_64, aarch64] 151 script: ['./build.sh $ARCH'] 152 ``` 153 154 produces `build:arch=x86_64` and `build:arch=aarch64`. 155 156 ### matrix 157 158 Expands over the cartesian product of its dimensions. Each value is available 159 as `$MATRIX_<NAME>` in upper case. Dimension order follows the file, so 160 generated names are predictable. 161 162 ```yaml 163 package: 164 matrix: 165 pkg: [musl, busybox] 166 mode: [debug, release] 167 ``` 168 169 produces four tasks, named `package:pkg=musl,mode=debug` and so on. 170 171 Matrix values become part of the task name, so they are limited to letters, 172 digits, underscore, dot and hyphen. Use `arch` rather than a matrix dimension 173 called `arch`. 174 175 Dependencies 176 ------------ 177 178 `needs` lists tasks that must succeed first. A cycle is rejected, naming the 179 cycle. 180 181 When a dependency shares a dimension with the dependent, it is matched on 182 that dimension rather than fanned out across it: 183 184 ```yaml 185 build: 186 arch: [x86_64, aarch64] 187 script: ['./build.sh $ARCH'] 188 189 package: 190 needs: [build] 191 arch: [x86_64, aarch64] 192 script: ['./package.sh $ARCH'] 193 194 publish: 195 needs: [package] 196 script: ['./publish.sh'] 197 ``` 198 199 `package:arch=x86_64` waits only for `build:arch=x86_64`. `publish` declares 200 no architecture, so it waits for every `package`. 201 202 Three forms are accepted: 203 204 ```yaml 205 needs: [build] # match shared dimensions 206 needs: ['build:arch=x86_64'] # one specific task 207 needs: 208 - task: build 209 match: all # every instance, ignore dimensions 210 ``` 211 212 If a shared dimension has no counterpart, that is an error rather than a 213 silently dropped dependency. 214 215 A dependency that fails normally causes its dependents, and everything 216 reachable from them, to be skipped. A dependency with `allow_failure: true` 217 satisfies its dependents whether it passes or not. 218 219 ### Running after something, whatever became of it 220 221 `after` waits for a task to finish without requiring it to have succeeded: 222 223 ```yaml 224 test: 225 script: ['./test.sh'] 226 227 report: 228 after: [test] 229 script: ['./report.sh'] 230 ``` 231 232 `report` runs once `test` has stopped, whether it passed, failed, or was 233 itself skipped because something it needed failed. This is what to reach for 234 with teardown, notification and result collection, which are worth running 235 precisely when the run went badly. 236 237 `after` takes the same three forms as `needs` and matches shared dimensions 238 the same way, because the matching decides which instance is meant and not 239 what waiting on it implies. Naming one task under both keys is the same as 240 naming it under `needs` alone. 241 242 Two things `after` deliberately does not change: 243 244 It does not rescue the run. A failed task still fails the job, whether or 245 not something ran after it. Use `allow_failure` on the task itself to say 246 that its failure does not count. 247 248 It does not supply anything. Artifacts from a task that failed may be 249 absent or half written, so a task that reads them wants `needs`. 250 251 Where `only` keeps a task out of a run, its `after` dependents still run: 252 there is nothing left to wait for, so the edge is dropped rather than the 253 dependent. This is the opposite of `needs`, which takes its dependents out 254 of the run with it. 255 256 Talking back from a script 257 -------------------------- 258 259 A script can write directives on its own output to hand a value to a later 260 task, or to say how it should be counted. Both take the same shape, a line 261 starting at column 0: 262 263 ``` 264 ::<command> <arguments>[::<data>] 265 ``` 266 267 The line is removed from the log before it is shipped, so a value written 268 this way does not appear in it. 269 270 ### Outputs 271 272 ```yaml 273 build: 274 script: 275 - 'make' 276 - 'echo "::set-output tag=$(git describe)::public"' 277 278 deploy: 279 script: ['./deploy.sh ${{ tasks.build.outputs.tag }}'] 280 ``` 281 282 `deploy` was never declared to need `build`. Reading its output says so, and 283 the dependency is recorded exactly as if `needs: [build]` had been written. 284 285 Outputs are masked unless stated otherwise, matching project variables: 286 287 ``` 288 ::set-output token=hunter2 # masked 289 ::set-output token=hunter2::masked # the same, said out loud 290 ::set-output tag=v1.2.3::public # shown in the API and the UI 291 ``` 292 293 A masked value is redacted from the rest of that task's log, and from the log 294 of every task that goes on to read it. It is never returned by the API and 295 shows as `[masked]` in the UI. Both kinds are encrypted at rest, exactly as 296 project variables are; `public` decides what may be shown again, not how it 297 is stored. 298 299 Masking is stricter here than it is for a project variable. A commit decides 300 what an output says, so a masked output is not readable back by anyone. A 301 project variable is set by the people who own the project, and they can read 302 it back. 303 304 Values are stored per task, and a value is only reachable through a reference 305 to the task that produced it, so two tasks may use the same output name 306 without colliding. A reference may be read anywhere `${{ }}` works except 307 `requires`, which chooses the worker before any task has run. 308 309 Naming a matrix task waits for every instance of it and looks the output up 310 across all of them, which is what a step collecting the whole matrix wants: 311 312 ```yaml 313 package: 314 arch: [x86_64, aarch64] 315 script: ['./build.sh', 'echo "::set-output path-${{ arch }}=/out/pkg.tar::public"'] 316 317 index: 318 script: ['./reindex.sh ${{ tasks.package.outputs.path-x86_64 }}'] 319 ``` 320 321 If the referencing task shares a dimension with the task it reads, the 322 reference narrows to its own instance, the same way `needs` does. 323 324 A reference to a value that was never produced fails the task before it is 325 handed to a worker, naming what is missing. It does not run with a hole where 326 the value should have been. 327 328 ### Choosing the outcome 329 330 ```yaml 331 package: 332 script: 333 - './build.sh' 334 - '[ -f out/pkg.tar ] || echo "::set-state skipped"' 335 ``` 336 337 `::set-state` takes `success`, `failed` or `skipped`, and overrides what the 338 exit code would have meant. It does not stop the script: the run continues, 339 and the last directive written is the one that counts. 340 341 A timeout, a cancellation, or a failure to start the container still fails 342 the task. Those are not outcomes a script gets to talk its way out of. 343 344 **A skip is not a failure.** A skipped task leaves the job green, and 345 everything that needed it still runs: the indexer at the end of a package 346 repository does its work whether or not this particular package was rebuilt. 347 What it does not do is invent values: a task reading an output that a skipped 348 task never produced fails on that missing output, which is the difference 349 between not needing the value and needing one that is not there. 350 351 Worker features 352 --------------- 353 354 `requires` names capabilities the worker must advertise. The conductor only 355 offers a task to a worker that advertises all of them. 356 357 ```yaml 358 package: 359 requires: [sign-key] 360 script: ['./sign.sh'] 361 ``` 362 363 What a feature provides is defined on the worker, not here: a mount, some 364 environment, a privileged container. This is how a signing key reaches a 365 build without the conductor ever holding it, and how a task asks for docker in 366 docker. 367 368 Services 369 -------- 370 371 Sidecar containers started alongside the task on the same network, reachable 372 by their alias. 373 374 ```yaml 375 test: 376 image: node:22 377 services: 378 - postgres:16 379 - image: docker:dind 380 alias: docker 381 env: 382 DOCKER_TLS_CERTDIR: '' 383 script: [npm test] 384 ``` 385 386 The alias defaults to the image name without registry, path or tag, so 387 `postgres:16` is reachable as `postgres`. Aliases must be unique within a task. 388 389 Artifacts 390 --------- 391 392 ```yaml 393 artifacts: 394 paths: 395 - dist/** 396 - build/*.tar.gz 397 when: on_success 398 expire: 30d 399 ``` 400 401 A bare list is shorthand for `paths`. Paths are relative to the workspace; 402 absolute paths are rejected. `when` is `on_success`, `on_failure` or 403 `always`. 404 405 `expire` sets an exact deadline for what this task produces, and overrides 406 the project's retention policy in both directions: sooner than the project 407 would delete, or later than it would keep. It also overrides the rule that 408 protects the most recent successful job, which is the point of it. A 409 node_modules tree is worth dropping within the hour whether or not its job 410 was green, and without that the largest artifacts are the ones kept 411 longest. 412 413 Leave it out and the project's policy applies, which is the usual case. 414 See [deployment.md](deployment.md) for what those policies are. 415 416 Artifacts are streamed rather than buffered, so size is bounded by where 417 they are stored rather than by the conductor's memory. Object storage 418 receives anything over 32 MiB as a multipart upload, which keeps memory 419 flat and lifts the five gigabyte limit that applies to a single request. 420 Disk images and other multi gigabyte output are fine; what they cost is 421 storage, not resident memory. 422 423 Working directory 424 ----------------- 425 426 ```yaml 427 version: 1 428 workdir: /usr/src/app 429 ``` 430 431 Where the tree is unpacked inside the task container, and the directory 432 every script starts in. It must be an absolute path of ordinary directory 433 names; it does not have to exist in the image, and is created as the tree 434 is unpacked. 435 436 Three answers, most specific first: this key, the project's setting, and 437 failing both `/work`. The repository has the last word because the path 438 belongs with the code: an image that expects to build in `/usr/src/app` 439 knows that, and whoever registered the project should not have to. 440 441 Nothing else is shared with the container. The tree arrives over the 442 docker API and artifacts leave the same way, so a worker needs no 443 directory in common with the tasks it jobs, and no storage of its own. 444 445 Environment 446 ----------- 447 448 `env` values must be scalars, and names must be valid environment variable 449 names. Task `env` is merged over `defaults.env`. 450 451 Injected automatically: 452 453 | Variable | Present when | 454 | ----------------- | -------------------------------- | 455 | `ARCH` | the task declares `arch` | 456 | `MATRIX_<NAME>` | for each matrix dimension | 457 458 Interpolation 459 ------------- 460 461 `${{ ... }}` is substituted in `image`, `script`, `env` values, `requires` 462 and `services`. Three expressions are available: 463 464 | Expression | Substituted | 465 | --------------------------------- | ---------------------- | 466 | `arch` | when the task is stored | 467 | `matrix.<name>` | when the task is stored | 468 | `tasks.<task>.outputs.<name>` | when the task is dispatched | 469 470 ```yaml 471 build: 472 arch: [x86_64] 473 matrix: 474 pkg: [musl] 475 image: 'builder:${{ arch }}' 476 script: ['./build.sh ${{ matrix.pkg }}'] 477 ``` 478 479 Referring to a dimension the task does not declare is an error. Inside 480 `script` the `$ARCH` and `$MATRIX_*` variables usually read better, since the 481 shell expands them; interpolation exists for the fields the shell never sees, 482 such as `image`. 483 484 The third is different in kind: its value does not exist until another task 485 has run, so it survives compilation and is filled in when a worker claims the 486 task. That is why it cannot appear in `requires`, which is read to decide 487 which worker to offer the task to. See [talking back from a 488 script](#talking-back-from-a-script). 489 490 Full example 491 ------------ 492 493 ```yaml 494 version: 1 495 496 defaults: 497 image: debian:bookworm-slim 498 timeout: 30m 499 500 tasks: 501 lint: 502 script: [make lint] 503 504 build: 505 arch: [x86_64, aarch64] 506 script: ['./mk/build.sh $ARCH'] 507 artifacts: 508 paths: [build/out/**] 509 expire: 7d 510 511 package: 512 needs: [build] 513 arch: [x86_64, aarch64] 514 matrix: 515 pkg: [musl, busybox] 516 requires: [sign-key] 517 script: ['./mk/package.sh $MATRIX_PKG $ARCH'] 518 artifacts: 519 paths: ['build/repo/$ARCH/*.apk'] 520 521 publish: 522 needs: [package, lint] 523 requires: [publish-key] 524 script: ['./mk/publish.sh'] 525 ``` 526 527 This yields nine tasks: one `lint`, two `build`, four `package` and one 528 `publish`. The two `build` tasks run at the same time as `lint`; each 529 `package` starts as soon as its own architecture's `build` finishes.