api.md (17070B)
1 HTTP API 2 ======== 3 4 Two words to get straight first, because everything below turns on them. 5 6 A **job** is one run of a pipeline. Creating it compiles the repository's 7 `.conductor.yml` at a commit, which produces one or more **tasks**. A task 8 is the standalone unit a worker runs: one container, one script, one 9 result. 10 11 A worker deals only in tasks. It is never told which project or job a task 12 belongs to, and the task id it receives carries no structure it could read 13 that out of. What it gets is a context and a script. 14 15 Everything the conductor does is under `/api/v1`. There is no second 16 surface: the application in `web/` is an ordinary client of what follows, 17 and the only page the conductor renders itself is the built-in login form. 18 19 Three kinds of credential reach it, all in `Authorization: Bearer`: 20 21 | Credential | Obtained by | Reaches | 22 | ------------------- | ---------------------------------- | --------------------------- | 23 | access token | the code flow, see `docs/auth.md` | everything a person may do | 24 | worker token | `POST /api/v1/workers` | the worker protocol only | 25 | project HMAC | a project's trigger secret | starting that project's jobs | 26 27 An access token is a JWT and a worker token is 64 hex characters, so the 28 two never parse as each other. The project HMAC is not a bearer credential 29 at all: it is `X-Hub-Signature-256` over the exact request body. 30 31 Note the two readings of `/api/v1/tasks/:task`. Asking after the task 32 itself is a public read; everything below it is the worker protocol and 33 needs a token. They are separate plugins for that reason, so the token 34 requirement cannot accidentally spread to the first or lapse on the rest. 35 36 Anonymous is not an error. A public project's jobs and tasks read without 37 any credential at all. 38 39 Responses 40 --------- 41 42 Every response carries `ok`, so success and failure are told apart by one 43 key rather than inferred from which others turned up. 44 45 ```json 46 { "ok": true, "data": { "...": "..." } } 47 ``` 48 49 ```json 50 { "ok": false, "error": { "name": "UnknownTask", "message": "No such task" } } 51 ``` 52 53 `error.name` is the identifier to branch on, and is stable. `error.message` 54 is for a person to read and may be reworded. Some errors add `error.details` 55 with machine-readable context; a 500 carries `details.request_id`, which is 56 the id the fault was logged under. 57 58 A resource the caller may not see returns 404 rather than 403, so absence 59 and denial are indistinguishable. 60 61 Responses holding a collection add `pagination`: 62 63 ```json 64 { 65 "ok": true, 66 "data": { "...": "..." }, 67 "pagination": { 68 "next": "https://ci.example.com/api/v1/tasks/<id>?offset=50&limit=50", 69 "offset": 0, 70 "total": 128 71 } 72 } 73 ``` 74 75 Ask with `?offset=` and `?limit=`. The limit defaults to 50 and is capped at 76 200; an offset or limit that cannot be read falls back rather than failing. 77 Follow `next` rather than building the URL, and stop when it is absent: 78 `next` is omitted on the last page rather than sent as null, as is `details` 79 when there is none. 80 81 Starting a job 82 -------------- 83 84 Two ways in, differing only in the shape of what they accept. Both take the 85 same signature, and both read the pipeline from the repository at the 86 commit rather than trusting anything the caller says about it. 87 88 ### From a push 89 90 ``` 91 POST /api/v1/projects/:project/trigger 92 ``` 93 94 Accepts the payload from `hooks/post-receive`, and the push events of 95 GitHub, Gitea and GitLab. Signed with the project's trigger secret: 96 `X-Hub-Signature-256: sha256=<hmac>` over the exact request body, or 97 `X-Gitlab-Token`. A session that may manage the project is accepted 98 instead, for callers driving the API rather than forwarding a hook. 99 100 ```json 101 { "sha": "<40 hex>", "base": "<40 hex or empty>", "ref": "refs/heads/main", "actor": "alice" } 102 ``` 103 104 A push that deletes a branch is reported as ignored rather than refused, 105 since there is nothing there to build. 106 107 ### Deliberately 108 109 ``` 110 POST /api/v1/projects/:project/jobs 111 ``` 112 113 For anything driving the conductor on purpose rather than forwarding a 114 hook. No payload archaeology: `sha` is required and must be a full commit 115 id, and a missing one is an error rather than something to infer. 116 117 ```json 118 { "sha": "<40 hex>", "ref": "refs/heads/main", "base": null, "actor": "alice", "trigger": "api" } 119 ``` 120 121 Only `sha` is required. `trigger` is recorded as-is and defaults to `api`, 122 so a job started this way stays distinguishable from one a hook forwarded. 123 124 Both return: 125 126 ```json 127 { "ok": true, "data": { "status": "created", "job_id": "m2y5rj7ykdm9sfbr54hf" } } 128 ``` 129 130 | Status | `error.name` | Meaning | 131 | ------ | -------------------------------------- | --------------------------------------------- | 132 | 200 | none | job created, or ignored for a branch deletion | 133 | 400 | `NoCommitInPayload`, `InvalidSha`, `ShaRequired`, `MalformedJson` | nothing usable to build | 134 | 401 | `InvalidSignature` | signature missing or wrong | 135 | 404 | `UnknownProject` | no such project | 136 | 409 | `ProjectDisabled` | project disabled | 137 138 The job is recorded and the response returns immediately; the pipeline is 139 compiled afterwards. A pipeline that will not compile therefore surfaces as 140 the job going to `failed` with the reason on it, not as an error here. 141 142 Reading a job back 143 ------------------ 144 145 ``` 146 GET /api/v1/jobs every job the caller may see 147 GET /api/v1/projects/:project/jobs one project's jobs 148 GET /api/v1/jobs/:job 149 GET /api/v1/projects/:project/jobs/:job 150 GET /api/v1/jobs/:job/tasks 151 POST /api/v1/jobs/:job/cancel 152 POST /api/v1/jobs/:job/retry 153 ``` 154 155 A job id is unique on its own, so naming the project is optional. Where it 156 is named it is asserted: a job of some other project is a 404 rather than 157 being quietly served. The lists take `?state=` alongside the paging. 158 159 ```json 160 { 161 "ok": true, 162 "data": { "id": "...", "number": 12, "state": "running", "ref": "refs/heads/main", 163 "head_sha": "...", "visibility": "private", "created_at": 1800000000000 } 164 } 165 ``` 166 167 Two ways to be allowed: the job is public, or the caller may manage the 168 project. 169 170 `/tasks` returns the tasks and the edges between them, which is what a 171 stage graph is drawn from: 172 173 ```json 174 { 175 "ok": true, 176 "data": { 177 "tasks": [ 178 { "id": "...", "name": "build:arch=x86_64", "state": "success", "exit_code": 0, 179 "arch": "x86_64", "attempt": 1, "worker_name": "builder-1" } 180 ], 181 "deps": [ 182 { "task_id": "...", "depends_on_id": "...", "kind": "needs" } 183 ] 184 }, 185 "pagination": { "offset": 0, "total": 6 } 186 } 187 ``` 188 189 `kind` is `needs` or `after`. An `after` edge orders a task without putting 190 it downstream of a failure. 191 192 `cancel` only applies to a running job and is a 409 otherwise. `retry` 193 starts a new job at the same commit and returns its id; the original stays 194 readable rather than being overwritten. 195 196 Reading a task back 197 ------------------- 198 199 ``` 200 GET /api/v1/tasks/:task 201 GET /api/v1/tasks/:task/log 202 GET /api/v1/tasks/:task/outputs 203 GET /api/v1/tasks/:task/artifacts 204 ``` 205 206 Each also answers under `/api/v1/projects/:project/tasks/:task`, where the 207 project is asserted: a guessed id cannot be read through whichever project 208 happens to be readable. A private task needs a session on both paths. 209 210 ```json 211 { 212 "ok": true, 213 "data": { 214 "task": { 215 "id": "m2y5t7deprrj9s6jvv4d", "name": "package:arch=x86_64,pkg=musl", 216 "base_name": "package", "state": "success", "arch": "x86_64", 217 "image": "debian:bookworm-slim", "attempt": 1, "max_attempts": 1, 218 "allow_failure": false, "exit_code": 0, "error": null, 219 "worker_name": "builder-1", "timeout": 3600, "workdir": "/work", 220 "needs": ["build:arch=x86_64"], "after": [], "matrix": { "pkg": "musl" }, 221 "artifact_paths": ["dist/**"], 222 "log_size": 4096, "log_expired": false, 223 "created_at": 1800000000000, "started_at": null, "finished_at": null, 224 "project_id": "demo", "job_id": "m2y5rj7ykdm9sfbr54hf", "job_number": 12, 225 "ref": "refs/heads/main", "sha": "<40 hex>", "visibility": "public" 226 } 227 } 228 } 229 ``` 230 231 The log, the outputs and the artifacts are each their own page, because 232 each grows without bound and none of them belongs inside one response. 233 234 ``` 235 GET /api/v1/tasks/:task/log?offset=0&limit=262144 236 ``` 237 238 ```json 239 { "ok": true, "data": { "offset": 0, "size": 4096, "data": "...", "expired": false } } 240 ``` 241 242 `size` is the whole log, so a tail asks from where it stopped and is given 243 only what is new. `expired` is true once retention has removed it, which is 244 a different answer from an empty log. 245 246 `artifact_paths` on the task is what the pipeline told it to collect; 247 `/artifacts` is what it actually stored. Each carries a ready download 248 `url`, a convenience rather than the permission: that route checks 249 visibility again for itself. 250 251 `/outputs` is what the task wrote with `::set-output`. A masked one reports 252 its name so the shape of a run is visible, and never its value. 253 254 There is no `env` and no `services` here, deliberately. Both can hold 255 project variables, and a service carries an environment of its own. The 256 `script` is left out for the same reason: a public job can be built from a 257 repository that is not public. A worker gets all three because it has to; 258 nobody else does. 259 260 Managing things 261 --------------- 262 263 All of this needs an access token. What it may reach follows from who the 264 token belongs to rather than from which URL was used: an administrator acts 265 on anything, anyone else on what they own. Something they may not see is 266 reported as missing, not refused. 267 268 ``` 269 GET /api/v1/users /users/:id PATCH, DELETE POST /users 270 GET /api/v1/projects /projects/:id PATCH, DELETE POST /projects 271 GET /api/v1/workers /workers/:id PATCH, DELETE POST /workers 272 GET /api/v1/sessions /sessions/:id DELETE 273 ``` 274 275 `me` is an ordinary user id that resolves to the caller, so `GET 276 /api/v1/users/me` is who am I. Listing users as a non-administrator returns 277 just themselves rather than a refusal. 278 279 Changing your own password takes `current_password` alongside the new one. 280 An administrator setting someone else's does not, which is what a reset is. 281 282 `repo_url` must be `https`, `ssh` or `git`, or an `scp`-style `user@host:path`. 283 Other schemes select a git transport rather than a location: `ext::` runs a 284 command, `file://` reads the conductor's own disk. 285 286 Two things are shown exactly once, in the response that creates them, and 287 are not readable afterwards: 288 289 ```json 290 { "ok": true, "data": { "id": "demo", "trigger_secret": "<64 hex>" } } 291 { "ok": true, "data": { "id": "...", "name": "builder-1", "token": "<64 hex>" } } 292 ``` 293 294 `POST /api/v1/projects/:id/trigger-secret` replaces the first. There is no 295 way to read either back; losing one means generating another. 296 297 Project retention takes three answers and they are all different. Absent 298 leaves the setting alone, `null` follows the server default, and `0` keeps 299 forever: 300 301 ``` 302 PATCH /api/v1/projects/:id { "artifact_keep_days": null, "log_keep_days": 0 } 303 ``` 304 305 Variables are per project. The listing carries names and flags only; a value 306 comes back one at a time, from its own endpoint, and every read is logged: 307 308 ``` 309 GET /api/v1/projects/:id/variables 310 GET /api/v1/projects/:id/variables/:name/value 311 PUT /api/v1/projects/:id/variables/:name { "value": "...", "masked": true } 312 DELETE /api/v1/projects/:id/variables/:name 313 ``` 314 315 All four are restricted to the project's owner and to administrators, so 316 being able to see a public project does not mean being able to read its 317 credentials. `masked` decides whether a value is redacted from job logs; it 318 does not decide who may read it. 319 320 A worker's machines and its recent tasks hang off it: 321 322 ``` 323 GET /api/v1/workers/:id/agents 324 GET /api/v1/workers/:id/tasks 325 ``` 326 327 Sessions are the revocation surface. Each row carries the browser and 328 address it was last used from, so a person can tell one from another. They 329 are recorded for that and nothing else: a session arriving from a new 330 address is ordinary, and is never refused for it. 331 332 The worker API 333 -------------- 334 335 Documented because a worker is a normal client of it, and anyone may write 336 another. Every call needs `Authorization: Bearer <worker token>`, and a 337 worker may only touch a task it currently holds. 338 339 | Method | Path | Purpose | 340 | ------ | -------------------------------------- | --------------------------------- | 341 | POST | `/api/v1/tasks/claim` | claim work, 204 when idle | 342 | GET | `/api/v1/tasks/:task/source.tar.gz` | the tree at the commit | 343 | POST | `/api/v1/tasks/:task/log` | append output | 344 | POST | `/api/v1/tasks/:task/artifacts` | upload one file | 345 | POST | `/api/v1/tasks/:task/heartbeat` | stay alive, learn of cancellation | 346 | POST | `/api/v1/tasks/:task/complete` | report the outcome | 347 348 Claim with a JSON body: 349 `{"arches":["x86_64","aarch64"],"features":["dind","sign-key"],"name":"my-worker"}`. 350 Only tasks whose architecture the worker offers and whose `requires` it 351 satisfies are handed out. A worker owned by a user is only offered that 352 user's projects. 353 354 `/api/v1/tasks/claim` is the only URL a worker needs to know. A claimed 355 task carries absolute URLs for every other call in `endpoints`, so the rest 356 of the surface is discovered rather than assembled: 357 358 ```json 359 { 360 "ok": true, 361 "data": { 362 "task": { 363 "id": "m2y5t7deprrj9s6jvv4d", 364 "name": "build:arch=x86_64", 365 "image": "debian:bookworm-slim", 366 "script": ["./build.sh $ARCH"], 367 "env": { "ARCH": "x86_64", "CONDUCTOR_PROJECT": "demo" }, 368 "requires": [], "services": [], "artifacts": { "paths": ["dist/**"] }, 369 "workdir": "/work", "timeout": 3600, "attempt": 1, 370 "sha": "<40 hex>", "ref": "refs/heads/main", 371 "masked": [], "heartbeat_interval": 30, 372 "endpoints": { 373 "source": "https://ci.example.com/api/v1/tasks/<id>/source.tar.gz", 374 "log": "https://ci.example.com/api/v1/tasks/<id>/log", 375 "artifact": "https://ci.example.com/api/v1/tasks/<id>/artifacts", 376 "heartbeat": "https://ci.example.com/api/v1/tasks/<id>/heartbeat", 377 "done": "https://ci.example.com/api/v1/tasks/<id>/complete" 378 } 379 } 380 } 381 } 382 ``` 383 384 There is no `project_id` and no `job_id` in that payload, deliberately. The 385 identifiers a build legitimately wants are in `env`, where they are opaque 386 strings the worker copies into the container without reading. 387 388 Log appends send `Content-Type: application/octet-stream` with 389 `X-Log-Offset` set to where the worker believes it is writing. An 390 overlapping chunk is trimmed, and a gap is refused with a 409 naming 391 `LogOffsetConflict` and carrying where to resume: 392 393 ```json 394 { 395 "ok": false, 396 "error": { 397 "name": "LogOffsetConflict", 398 "message": "The log chunk does not continue from the end of the log", 399 "details": { "expected_offset": 4096 } 400 } 401 } 402 ``` 403 404 Artifact uploads send the bytes with `X-Artifact-Path` and a 405 `Content-Length`. The path is sanitised, so traversal is stripped rather 406 than honoured. Bodies are streamed and may be arbitrarily large; a body 407 that does not match its declared length is rejected. 408 409 `complete` reports the outcome, and is sent a body rather than given one. 410 Requests are not enveloped; only responses are: 411 412 ```json 413 { 414 "success": true, "state": "success", "exit_code": 0, "error": null, 415 "outputs": [{ "name": "tag", "value": "v1.2.3", "masked": false }] 416 } 417 ``` 418 419 `state` is one of `success`, `failed` or `skipped`, and is what the task 420 wrote with `::set-state` if it wrote one. It takes precedence over 421 `success`, which is kept for workers that do not send a state. Names and 422 sizes in `outputs` are validated again here rather than trusted from the 423 worker, and values are encrypted before they are stored. 424 425 A failure with attempts remaining requeues the task, discarding any outputs 426 that attempt produced; otherwise everything that `needs` it, directly or 427 transitively, is skipped. Tasks that only name it under `after` are released 428 instead, since they asked to follow it rather than to be fed by it. 429 430 A task that reports `skipped` is not a failure: the job can still end 431 `success`, and tasks that `needs` it still run. One of them reading an output 432 it never produced fails on that, before it is handed to a worker. 433 434 Artifacts 435 --------- 436 437 ``` 438 GET /api/v1/projects/:project/jobs/:job/tasks/:task/artifacts/:artifact 439 ``` 440 441 The one read endpoint that exists for a browser, so the task page can link 442 straight at a build's output. The project, job and task in the path must 443 all agree with the artifact. A public job is downloadable by anyone; a 444 private one by its owner or an administrator. The response is the bytes, or 445 a redirect to the object store when it can presign.