worker.md (6716B)
1 Running a worker 2 ================ 3 4 A worker polls a conductor for tasks and runs them in containers. It needs no 5 inbound connectivity, so it can sit behind NAT, and it needs no repository 6 credentials, because the conductor serves the source for the exact commit a 7 task was handed. 8 9 The worker has no npm dependencies and keeps no state. Copying 10 `src/worker` onto a host with node and a container runtime is enough. 11 12 Requirements 13 ------------ 14 15 - node 24 or newer 16 - a docker compatible CLI: docker, podman or nerdctl 17 18 That is the whole list. A task's tree is unpacked into the task's own 19 container over the docker API, so the worker needs no `tar` of its own, 20 and it never touches a repository, so it needs no `git` either. 21 22 Getting a token 23 --------------- 24 25 Ask whoever runs the conductor for a worker token. They create one with: 26 27 ```sh 28 node src/admin-cli.js token:add friends-pi 29 ``` 30 31 The token is shown once and stored only as a hash, so it cannot be recovered 32 later. If it is lost, issue a new one and remove the old. 33 34 Configuration 35 ------------- 36 37 JSON is the native format. YAML also works when the `yaml` package is 38 installed, which it is in the published image. 39 40 ```json 41 { 42 "conductor_url": "https://ci.example.com", 43 "name": "friends-pi", 44 "token_file": "/run/secrets/worker_token", 45 "arches": ["aarch64"], 46 "concurrency": 2, 47 "features": { 48 "dind": { "privileged": true }, 49 "sign-key": { 50 "mounts": ["/srv/keys/build.rsa:/keys/build.rsa:ro"], 51 "env": { "UNOS_SIGN_KEY": "/keys/build.rsa" } 52 } 53 } 54 } 55 ``` 56 57 ```sh 58 node src/worker/agent.js --config worker.json 59 ``` 60 61 | Key | Default | Description | 62 | ---------------- | -------------------- | ----------------------------------------------- | 63 | `conductor_url` | `http://127.0.0.1:8080` | Base URL of the conductor. | 64 | `name` | hostname | Shown against tasks this worker ran. | 65 | `token` | none | The worker token, inline. | 66 | `token_file` | none | Path to a file holding the token. Preferred. | 67 | `arches` | `[]` | Architectures offered. Empty means no `arch` tasks, only tasks that declare none. | 68 | `features` | `{}` | Capabilities offered, see below. | 69 | `concurrency` | `1` | Tasks run at once. | 70 | `poll_interval` | `5` | Seconds between polls when idle. | 71 | `docker` | `docker` | Runtime CLI. Set to `podman` or `nerdctl`. | 72 | `shell` | `sh` | Shell used to run a task script in its image. | 73 74 There is nothing here about where tasks run or what they are given, which 75 is deliberate: that is a property of the build and is decided by the 76 pipeline or the project, not by the machine that happens to run it. 77 78 Every key can also be set from the environment: `CONDUCTOR_URL`, 79 `CONDUCTOR_WORKER_TOKEN`, `CONDUCTOR_WORKER_TOKEN_FILE`, 80 `CONDUCTOR_WORKER_ARCHES`, `CONDUCTOR_WORKER_CONCURRENCY`, 81 `CONDUCTOR_WORKER_DOCKER`, and so on. The environment wins over the file. 82 83 Features 84 -------- 85 86 A feature is a name this worker advertises, together with whatever local 87 resources a task asking for it should receive. The conductor only ever learns 88 the name. It will not send a task to a worker that does not advertise every 89 feature the task's `requires` lists. 90 91 ```json 92 "features": { 93 "sign-key": { 94 "mounts": ["/srv/keys/build.rsa:/keys/build.rsa:ro"], 95 "env": { "SIGN_KEY": "/keys/build.rsa" }, 96 "devices": [], 97 "privileged": false 98 } 99 } 100 ``` 101 102 A pipeline then asks for it by name: 103 104 ```yaml 105 package: 106 requires: [sign-key] 107 script: ['./sign.sh'] 108 ``` 109 110 This is how a signing key reaches a build without the conductor ever holding 111 it. The key stays on the machine that owns it, and only tasks that explicitly 112 ask for it are ever scheduled there. 113 114 `privileged: true` is what a docker in docker service needs. Grant it only 115 to features you intend to be privileged, since it removes the isolation 116 between the task and the host. 117 118 What a task gets 119 --------------- 120 121 Each task runs in its own container, on its own network: 122 123 - the repository tree at the task's commit, unpacked into the working 124 directory, which is `/work` unless the pipeline or the project says 125 otherwise 126 - that directory as the working directory 127 - the task's `env`, plus `ARCH` and `MATRIX_*` for a fanned out task 128 - `CONDUCTOR_PROJECT`, `CONDUCTOR_JOB_ID`, `CONDUCTOR_JOB_NUMBER`, 129 `CONDUCTOR_TASK`, `CONDUCTOR_TASK_ID`, `CONDUCTOR_SHA`, `CONDUCTOR_REF` 130 and `CONDUCTOR_ATTEMPT`, which name the project and job the worker 131 itself is never told about: they are opaque strings it copies in 132 - any `services`, reachable by their alias on the task network 133 - mounts and environment from the features it requires 134 135 Output is streamed to the conductor as it happens. Artifacts are read 136 back out of the container when the task finishes, and uploaded. 137 138 Nothing else is shared. The worker mounts no directory of its own into a 139 task: the tree goes in over the docker API and the artifacts come back the 140 same way. That is why a worker needs no volumes, no matching paths 141 between itself and the host, and no storage at all. 142 143 Operational notes 144 ----------------- 145 146 **Cleanup.** Containers and networks are removed whatever the outcome. 147 There is no workspace to clean up, and so no trouble with a task having 148 written files as root that the worker then cannot delete: everything the 149 task wrote lived in its container and went with it. 150 151 **Shutdown.** On SIGINT or SIGTERM the worker stops polling and lets running 152 tasks finish. A second signal exits immediately, and the conductor will 153 eventually reap whatever was left running. 154 155 **Failures.** A worker that stops reporting has its tasks requeued or failed 156 by the conductor after `scheduler.heartbeat_timeout`. Losing a worker never 157 strands a job. 158 159 **Log loss.** If the conductor cannot be reached while a task runs, log output 160 for that window is dropped rather than failing the task. The task's outcome is 161 still reported. 162 163 Trust 164 ----- 165 166 A worker is trusted with the source of the commits it builds and with 167 whatever its features grant it. It is not trusted with anything else: it 168 holds no repository credentials, it cannot read other refs, it cannot reach 169 another project's tasks, and it cannot write to storage except by uploading 170 artifacts for a task it currently holds. 171 172 Running a task means running code from the repository. Only accept work from 173 a conductor whose projects you are willing to execute, and keep features 174 narrow.