api.md (10654B)
1 HTTP API 2 ======== 3 4 Two words to get straight first, because everything below turns on them. 5 6 A **job** is one run of a pipeline. Creating it compiles the repository's 7 `.conductor.yml` at a commit, which produces one or more **tasks**. A task 8 is the standalone unit a worker runs: one container, one script, one 9 result. 10 11 A worker deals only in tasks. It is never told which project or job a task 12 belongs to, and the task id it receives carries no structure it could read 13 that out of. What it gets is a context and a script. 14 15 The conductor exposes five HTTP surfaces, all under `/api/v1`. Everything 16 else, including signing in and managing projects, workers and users, 17 happens in the interface. 18 19 | Surface | Authentication | 20 | --------------------------------------------- | --------------------- | 21 | `POST /api/v1/projects/:project/trigger` | per project HMAC | 22 | `POST /api/v1/projects/:project/jobs` | per project HMAC | 23 | `GET /api/v1/projects/:project/jobs/:job` | HMAC, session, or anonymous for a public job | 24 | `GET /api/v1/tasks/:task` | session, or anonymous for a public task | 25 | `GET /api/v1/projects/:project/tasks/:task` | HMAC, session, or anonymous for a public task | 26 | `POST /api/v1/tasks/claim`, `/tasks/:task/...` | worker token | 27 | `GET /api/v1/projects/.../artifacts/...` | session, or anonymous for a public job | 28 29 Note the two readings of `/api/v1/tasks/:task`. Asking after the task 30 itself is a public read; everything below it is the worker protocol and 31 needs a token. They are separate plugins for that reason, so the token 32 requirement cannot accidentally spread to the first or lapse on the rest. 33 34 Errors are `{"error": "..."}` with a meaningful status. A resource the 35 caller may not see returns 404 rather than 403, so absence and denial are 36 indistinguishable. 37 38 Starting a job 39 -------------- 40 41 Two ways in, differing only in the shape of what they accept. Both take the 42 same signature, and both read the pipeline from the repository at the 43 commit rather than trusting anything the caller says about it. 44 45 ### From a push 46 47 ``` 48 POST /api/v1/projects/:project/trigger 49 ``` 50 51 Accepts the payload from `hooks/post-receive`, and the push events of 52 GitHub, Gitea and GitLab. Signed with the project's trigger secret: 53 `X-Hub-Signature-256: sha256=<hmac>` over the exact request body, or 54 `X-Gitlab-Token`. 55 56 ```json 57 { "sha": "<40 hex>", "base": "<40 hex or empty>", "ref": "refs/heads/main", "actor": "alice" } 58 ``` 59 60 A push that deletes a branch is reported as ignored rather than refused, 61 since there is nothing there to build. 62 63 ### Deliberately 64 65 ``` 66 POST /api/v1/projects/:project/jobs 67 ``` 68 69 For anything driving the conductor on purpose rather than forwarding a 70 hook. No payload archaeology: `sha` is required and must be a full commit 71 id, and a missing one is an error rather than something to infer. 72 73 ```json 74 { "sha": "<40 hex>", "ref": "refs/heads/main", "base": null, "actor": "alice", "trigger": "api" } 75 ``` 76 77 Only `sha` is required. `trigger` is recorded as-is and defaults to `api`, 78 so a job started this way stays distinguishable from one a hook forwarded. 79 80 Both return: 81 82 ```json 83 { "status": "created", "job_id": "m2y5rj7ykdm9sfbr54hf", "tasks": 6 } 84 ``` 85 86 | Status | Meaning | 87 | ------ | ---------------------------------------------- | 88 | 200 | job created, or ignored for a branch deletion | 89 | 400 | no usable commit in the payload | 90 | 401 | signature missing or wrong | 91 | 404 | no such project | 92 | 409 | project disabled | 93 | 422 | the pipeline at that commit is invalid | 94 95 A 422 carries `problems`, an array of `{path, message}`, so a broken 96 pipeline says which key is wrong rather than only that something is. 97 98 Reading a job back 99 ------------------ 100 101 ``` 102 GET /api/v1/projects/:project/jobs/:job 103 ``` 104 105 Returns the job and the state of every task in it. 106 107 ```json 108 { 109 "job": { "id": "...", "number": 12, "state": "running", "ref": "refs/heads/main", 110 "head_sha": "...", "visibility": "private", "created_at": 1800000000000 }, 111 "tasks": [ 112 { "id": "...", "name": "build:arch=x86_64", "state": "success", "exit_code": 0, 113 "arch": "x86_64", "attempt": 1, "worker_name": "builder-1" } 114 ] 115 } 116 ``` 117 118 Three ways to be allowed: the job is public, the caller holds the project's 119 trigger secret, or the caller is signed in and may manage the project. One 120 credential therefore covers starting a build and watching it. 121 122 Reading a task back 123 ------------------- 124 125 ``` 126 GET /api/v1/tasks/:task 127 GET /api/v1/projects/:project/tasks/:task 128 ``` 129 130 A task's state and the artifacts it stored. Both paths return the same 131 body. A task id is unique on its own, so naming the project is optional; 132 doing so only widens what may reach a private task, because there is then a 133 trigger secret to check against. 134 135 | | public task | trigger secret | session that may manage the project | 136 | --- | --- | --- | --- | 137 | `/tasks/:task` | yes | not applicable | yes | 138 | `/projects/:project/tasks/:task` | yes | yes | yes | 139 140 A signature on the bare path proves nothing, since there is no project in 141 it to look a secret up from. On the scoped path the project must be the one 142 the task belongs to, so a guessed id cannot be read through some other 143 project that happens to be readable. Signing a GET means an HMAC over an 144 empty body. 145 146 ```json 147 { 148 "task": { 149 "id": "m2y5t7deprrj9s6jvv4d", "name": "package:arch=x86_64,pkg=musl", 150 "base_name": "package", "state": "success", "arch": "x86_64", 151 "image": "debian:bookworm-slim", "attempt": 1, "max_attempts": 1, 152 "allow_failure": false, "exit_code": 0, "error": null, 153 "worker_name": "builder-1", "timeout": 3600, "workdir": "/work", 154 "needs": ["build:arch=x86_64"], "matrix": { "pkg": "musl" }, 155 "artifact_paths": ["dist/**"], "log_size": 4096, 156 "created_at": 1800000000000, "started_at": null, "finished_at": null, 157 "project_id": "demo", "job_id": "m2y5rj7ykdm9sfbr54hf", "job_number": 12, 158 "ref": "refs/heads/main", "sha": "<40 hex>", "visibility": "public" 159 }, 160 "artifacts": [ 161 { "id": "...", "path": "dist/app", "size": 1048576, "sha256": "<64 hex>", 162 "created_at": 1800000000000, "expires_at": null, 163 "url": "https://ci.example.com/api/v1/projects/demo/jobs/<job>/tasks/<task>/artifacts/<id>" } 164 ] 165 } 166 ``` 167 168 `artifact_paths` is what the pipeline told the task to collect; `artifacts` 169 is what it actually stored. Each carries a ready download `url`, which is a 170 convenience rather than the permission: that route checks visibility again 171 for itself. 172 173 There is no `env` and no `services` here, deliberately. Both can hold 174 project variables, and a service carries an environment of its own. The 175 `script` is left out for the same reason: a public job can be built from a 176 repository that is not public. A worker gets all three because it has to; 177 nobody else does. 178 179 The worker API 180 -------------- 181 182 Documented because a worker is a normal client of it, and anyone may write 183 another. Every call needs `Authorization: Bearer <worker token>`, and a 184 worker may only touch a task it currently holds. 185 186 | Method | Path | Purpose | 187 | ------ | -------------------------------------- | --------------------------------- | 188 | POST | `/api/v1/tasks/claim` | claim work, 204 when idle | 189 | GET | `/api/v1/tasks/:task/source.tar.gz` | the tree at the commit | 190 | POST | `/api/v1/tasks/:task/log` | append output | 191 | POST | `/api/v1/tasks/:task/artifacts` | upload one file | 192 | POST | `/api/v1/tasks/:task/heartbeat` | stay alive, learn of cancellation | 193 | POST | `/api/v1/tasks/:task/complete` | report the outcome | 194 195 Claim with a JSON body: 196 `{"arches":["x86_64","aarch64"],"features":["dind","sign-key"],"name":"my-worker"}`. 197 Only tasks whose architecture the worker offers and whose `requires` it 198 satisfies are handed out. A worker owned by a user is only offered that 199 user's projects. 200 201 `/api/v1/tasks/claim` is the only URL a worker needs to know. A claimed 202 task carries absolute URLs for every other call in `endpoints`, so the rest 203 of the surface is discovered rather than assembled: 204 205 ```json 206 { 207 "task": { 208 "id": "m2y5t7deprrj9s6jvv4d", 209 "name": "build:arch=x86_64", 210 "image": "debian:bookworm-slim", 211 "script": ["./build.sh $ARCH"], 212 "env": { "ARCH": "x86_64", "CONDUCTOR_PROJECT": "demo" }, 213 "requires": [], "services": [], "artifacts": { "paths": ["dist/**"] }, 214 "workdir": "/work", "timeout": 3600, "attempt": 1, 215 "sha": "<40 hex>", "ref": "refs/heads/main", 216 "masked": [], "heartbeat_interval": 30, 217 "endpoints": { 218 "source": "https://ci.example.com/api/v1/tasks/<id>/source.tar.gz", 219 "log": "https://ci.example.com/api/v1/tasks/<id>/log", 220 "artifact": "https://ci.example.com/api/v1/tasks/<id>/artifacts", 221 "heartbeat": "https://ci.example.com/api/v1/tasks/<id>/heartbeat", 222 "done": "https://ci.example.com/api/v1/tasks/<id>/complete" 223 } 224 } 225 } 226 ``` 227 228 There is no `project_id` and no `job_id` in that payload, deliberately. The 229 identifiers a build legitimately wants are in `env`, where they are opaque 230 strings the worker copies into the container without reading. 231 232 Log appends send `Content-Type: application/octet-stream` with 233 `X-Log-Offset` set to where the worker believes it is writing. An 234 overlapping chunk is trimmed and a gap is refused with 409 and 235 `expected_offset`, which makes a retry after a dropped connection safe. 236 237 Artifact uploads send the bytes with `X-Artifact-Path` and a 238 `Content-Length`. The path is sanitised, so traversal is stripped rather 239 than honoured. Bodies are streamed and may be arbitrarily large; a body 240 that does not match its declared length is rejected. 241 242 ```json 243 { "success": true, "exit_code": 0, "error": null } 244 ``` 245 246 `complete` reports the outcome. A failure with attempts remaining requeues 247 the task; otherwise everything downstream of it is skipped. 248 249 Artifacts 250 --------- 251 252 ``` 253 GET /api/v1/projects/:project/jobs/:job/tasks/:task/artifacts/:artifact 254 ``` 255 256 The one read endpoint that exists for a browser, so the task page can link 257 straight at a build's output. The project, job and task in the path must 258 all agree with the artifact. A public job is downloadable by anyone; a 259 private one by its owner or an administrator. The response is the bytes, or 260 a redirect to the object store when it can presign.