pipeline.md (11955B)
1 .conductor.yml reference 2 ======================== 3 4 The pipeline file is read from the repository at the commit being built, so 5 it is versioned with the code it builds. The conductor never executes 6 anything from the repository while parsing it; the file is data. 7 8 Every unknown key is an error rather than a warning, and all problems in a 9 file are reported together. 10 11 Top level 12 --------- 13 14 | Key | Required | Description | 15 | ------------ | -------- | --------------------------------------------------- | 16 | `version` | yes | Must be `1`. | 17 | `visibility` | no | `public` or `private`. Overrides the project. | 18 | `workdir` | no | Absolute path tasks run in. Overrides the project. | 19 | `defaults` | no | Values inherited by every task. | 20 | `tasks` | yes | Mapping of task name to task. | 21 22 The file may be named `.conductor.yml` or `.conductor.yaml`; both are 23 looked for unless the project was pointed at some other path explicitly. 24 Carrying both at once is an error rather than a coin toss, since two 25 pipelines in one repository will disagree sooner or later. 26 27 This key was called `jobs` before 0.3.0. A job is now the pipeline run 28 that these tasks belong to, and a file still using the old spelling is 29 refused with a message saying so. 30 31 Task names may contain letters, digits, underscore, dot and hyphen, and must 32 start with a letter or digit. 33 34 ```yaml 35 version: 1 36 defaults: 37 image: debian:bookworm-slim 38 tasks: 39 test: 40 script: [make test] 41 ``` 42 43 Visibility 44 ---------- 45 46 Jobs are private unless something says otherwise, and the repository has the 47 final word on its own results: 48 49 ```yaml 50 version: 1 51 visibility: public 52 ``` 53 54 Unset means the project's setting stands. The value is recorded on each job 55 as it is created, so turning a repository private stops exposing future jobs 56 without retroactively hiding, or revealing, earlier ones. 57 58 A public job, its logs and its artifacts are readable by anyone. A private 59 job is readable by the project's owner and by administrators. 60 61 Tasks 62 ---- 63 64 | Key | Type | Default | Inheritable | 65 | --------------- | ----------------- | -------------- | ----------- | 66 | `image` | string | none, required | yes | 67 | `script` | list of strings | none, required | no | 68 | `needs` | list | `[]` | no | 69 | `arch` | string or list | none | yes | 70 | `matrix` | mapping | none | no | 71 | `requires` | list of strings | `[]` | yes | 72 | `services` | list | `[]` | yes | 73 | `env` | mapping | `{}` | merged | 74 | `artifacts` | list or mapping | none | no | 75 | `allow_failure` | boolean | `false` | yes | 76 | `timeout` | duration | server default | yes | 77 | `max_attempts` | integer, 1 to 10 | `1` | yes | 78 | `only` | mapping | none | no | 79 80 `image` and `script` are the only required fields, and `image` may come from 81 `defaults`. 82 83 ### script 84 85 A list of shell commands, run in order in the task's container. The task fails 86 on the first command that exits non-zero. 87 88 Take care with YAML scalars. Unquoted `true`, `false`, `yes`, `no` and `on` 89 parse as booleans, not strings, and are rejected: 90 91 ```yaml 92 script: ['true'] # correct 93 script: [true] # rejected, this is a boolean 94 ``` 95 96 ### timeout 97 98 Accepts whole seconds, or a duration with units `s`, `m`, `h`, `d` and `w`, 99 optionally combined: `90`, `90s`, `30m`, `1h`, `1h30m`. A task timeout may not 100 exceed seven days; `artifacts.expire` may not exceed a year. 101 102 Fanning out 103 ----------- 104 105 ### only 106 107 Restricts a task to certain refs. Without it a task runs on every push. 108 109 ```yaml 110 publish: 111 needs: [build] 112 only: 113 refs: [refs/heads/main, 'refs/tags/v*'] 114 script: 115 - ./deploy/publish.sh 116 ``` 117 118 Patterns match the whole ref, so `refs/heads/main` rather than `main`. 119 A `*` stands for any run of characters; everything else is literal, so a 120 dot is a dot rather than any character. 121 122 A task that does not match is left out of the job entirely, not recorded 123 as skipped. A skipped task means a dependency collapsed and fails the job, 124 which is the wrong reading for a publish step that was never meant to run 125 on this branch. 126 127 Anything depending on an excluded task is excluded with it. Dropping the 128 dependency instead would let a task run without something it declared it 129 needed, which is the worse surprise: 130 131 ```yaml 132 # On a branch, none of these three run. 133 publish: 134 only: { refs: [refs/heads/main] } 135 script: ['make publish'] 136 announce: 137 needs: [publish] 138 script: ['make announce'] 139 ``` 140 141 A job triggered without a ref matches no pattern, so restricted tasks stay 142 out rather than being handed a ref they were never written for. 143 144 The rule is not inheritable from `defaults`. A restriction that silently 145 applied to every task is hard to spot when the symptom is an empty job. 146 147 ### arch 148 149 Expands the task into one task per architecture, and restricts each to a worker 150 that offers it. The value is available to the script as `$ARCH`. 151 152 ```yaml 153 build: 154 arch: [x86_64, aarch64] 155 script: ['./build.sh $ARCH'] 156 ``` 157 158 produces `build:arch=x86_64` and `build:arch=aarch64`. 159 160 ### matrix 161 162 Expands over the cartesian product of its dimensions. Each value is available 163 as `$MATRIX_<NAME>` in upper case. Dimension order follows the file, so 164 generated names are predictable. 165 166 ```yaml 167 package: 168 matrix: 169 pkg: [musl, busybox] 170 mode: [debug, release] 171 ``` 172 173 produces four tasks, named `package:pkg=musl,mode=debug` and so on. 174 175 Matrix values become part of the task name, so they are limited to letters, 176 digits, underscore, dot and hyphen. Use `arch` rather than a matrix dimension 177 called `arch`. 178 179 Dependencies 180 ------------ 181 182 `needs` lists tasks that must succeed first. A cycle is rejected, naming the 183 cycle. 184 185 When a dependency shares a dimension with the dependent, it is matched on 186 that dimension rather than fanned out across it: 187 188 ```yaml 189 build: 190 arch: [x86_64, aarch64] 191 script: ['./build.sh $ARCH'] 192 193 package: 194 needs: [build] 195 arch: [x86_64, aarch64] 196 script: ['./package.sh $ARCH'] 197 198 publish: 199 needs: [package] 200 script: ['./publish.sh'] 201 ``` 202 203 `package:arch=x86_64` waits only for `build:arch=x86_64`. `publish` declares 204 no architecture, so it waits for every `package`. 205 206 Three forms are accepted: 207 208 ```yaml 209 needs: [build] # match shared dimensions 210 needs: ['build:arch=x86_64'] # one specific task 211 needs: 212 - task: build 213 match: all # every instance, ignore dimensions 214 ``` 215 216 If a shared dimension has no counterpart, that is an error rather than a 217 silently dropped dependency. 218 219 A dependency that fails normally causes its dependents, and everything 220 reachable from them, to be skipped. A dependency with `allow_failure: true` 221 satisfies its dependents whether it passes or not. 222 223 Worker features 224 --------------- 225 226 `requires` names capabilities the worker must advertise. The conductor only 227 offers a task to a worker that advertises all of them. 228 229 ```yaml 230 package: 231 requires: [sign-key] 232 script: ['./sign.sh'] 233 ``` 234 235 What a feature provides is defined on the worker, not here: a mount, some 236 environment, a privileged container. This is how a signing key reaches a 237 build without the conductor ever holding it, and how a task asks for docker in 238 docker. 239 240 Services 241 -------- 242 243 Sidecar containers started alongside the task on the same network, reachable 244 by their alias. 245 246 ```yaml 247 test: 248 image: node:22 249 services: 250 - postgres:16 251 - image: docker:dind 252 alias: docker 253 env: 254 DOCKER_TLS_CERTDIR: '' 255 script: [npm test] 256 ``` 257 258 The alias defaults to the image name without registry, path or tag, so 259 `postgres:16` is reachable as `postgres`. Aliases must be unique within a task. 260 261 Artifacts 262 --------- 263 264 ```yaml 265 artifacts: 266 paths: 267 - dist/** 268 - build/*.tar.gz 269 when: on_success 270 expire: 30d 271 ``` 272 273 A bare list is shorthand for `paths`. Paths are relative to the workspace; 274 absolute paths are rejected. `when` is `on_success`, `on_failure` or 275 `always`. 276 277 `expire` sets an exact deadline for what this task produces, and overrides 278 the project's retention policy in both directions: sooner than the project 279 would delete, or later than it would keep. It also overrides the rule that 280 protects the most recent successful job, which is the point of it. A 281 node_modules tree is worth dropping within the hour whether or not its job 282 was green, and without that the largest artifacts are the ones kept 283 longest. 284 285 Leave it out and the project's policy applies, which is the usual case. 286 See [deployment.md](deployment.md) for what those policies are. 287 288 Artifacts are streamed rather than buffered, so size is bounded by where 289 they are stored rather than by the conductor's memory. Object storage 290 receives anything over 32 MiB as a multipart upload, which keeps memory 291 flat and lifts the five gigabyte limit that applies to a single request. 292 Disk images and other multi gigabyte output are fine; what they cost is 293 storage, not resident memory. 294 295 Working directory 296 ----------------- 297 298 ```yaml 299 version: 1 300 workdir: /usr/src/app 301 ``` 302 303 Where the tree is unpacked inside the task container, and the directory 304 every script starts in. It must be an absolute path of ordinary directory 305 names; it does not have to exist in the image, and is created as the tree 306 is unpacked. 307 308 Three answers, most specific first: this key, the project's setting, and 309 failing both `/work`. The repository has the last word because the path 310 belongs with the code: an image that expects to build in `/usr/src/app` 311 knows that, and whoever registered the project should not have to. 312 313 Nothing else is shared with the container. The tree arrives over the 314 docker API and artifacts leave the same way, so a worker needs no 315 directory in common with the tasks it jobs, and no storage of its own. 316 317 Environment 318 ----------- 319 320 `env` values must be scalars, and names must be valid environment variable 321 names. Task `env` is merged over `defaults.env`. 322 323 Injected automatically: 324 325 | Variable | Present when | 326 | ----------------- | -------------------------------- | 327 | `ARCH` | the task declares `arch` | 328 | `MATRIX_<NAME>` | for each matrix dimension | 329 330 Interpolation 331 ------------- 332 333 `${{ ... }}` is substituted before the task is stored, in `image`, `script`, 334 `env` values, `requires` and `services`. Two expressions are available: 335 `arch` and `matrix.<name>`. 336 337 ```yaml 338 build: 339 arch: [x86_64] 340 matrix: 341 pkg: [musl] 342 image: 'builder:${{ arch }}' 343 script: ['./build.sh ${{ matrix.pkg }}'] 344 ``` 345 346 Referring to a dimension the task does not declare is an error. Inside 347 `script` the `$ARCH` and `$MATRIX_*` variables usually read better, since the 348 shell expands them; interpolation exists for the fields the shell never sees, 349 such as `image`. 350 351 Full example 352 ------------ 353 354 ```yaml 355 version: 1 356 357 defaults: 358 image: debian:bookworm-slim 359 timeout: 30m 360 361 tasks: 362 lint: 363 script: [make lint] 364 365 build: 366 arch: [x86_64, aarch64] 367 script: ['./mk/build.sh $ARCH'] 368 artifacts: 369 paths: [build/out/**] 370 expire: 7d 371 372 package: 373 needs: [build] 374 arch: [x86_64, aarch64] 375 matrix: 376 pkg: [musl, busybox] 377 requires: [sign-key] 378 script: ['./mk/package.sh $MATRIX_PKG $ARCH'] 379 artifacts: 380 paths: ['build/repo/$ARCH/*.apk'] 381 382 publish: 383 needs: [package, lint] 384 requires: [publish-key] 385 script: ['./mk/publish.sh'] 386 ``` 387 388 This yields nine tasks: one `lint`, two `build`, four `package` and one 389 `publish`. The two `build` tasks run at the same time as `lint`; each 390 `package` starts as soon as its own architecture's `build` finishes.