worker.md (7178B)
1 Running a worker 2 ================ 3 4 A worker polls a conductor for tasks and runs them in containers. It needs no 5 inbound connectivity, so it can sit behind NAT, and it needs no repository 6 credentials, because the conductor serves the source for the exact commit a 7 task was handed. 8 9 The worker has no npm dependencies and keeps no state. Copying 10 `src/worker` onto a host with node and a container runtime is enough. 11 12 Requirements 13 ------------ 14 15 - node 24 or newer 16 - a docker compatible CLI: docker, podman or nerdctl 17 18 That is the whole list. A task's tree is unpacked into the task's own 19 container over the docker API, so the worker needs no `tar` of its own, 20 and it never touches a repository, so it needs no `git` either. 21 22 Getting a token 23 --------------- 24 25 Ask whoever runs the conductor for a worker token. They create one with: 26 27 ```sh 28 node src/admin-cli.js token:add friends-pi 29 ``` 30 31 The token is shown once and stored only as a hash, so it cannot be recovered 32 later. If it is lost, issue a new one and remove the old. 33 34 Configuration 35 ------------- 36 37 JSON is the native format. YAML also works when the `yaml` package is 38 installed, which it is in the published image. 39 40 ```json 41 { 42 "conductor_url": "https://ci.example.com", 43 "name": "friends-pi", 44 "token_file": "/run/secrets/worker_token", 45 "arches": ["aarch64"], 46 "concurrency": 2, 47 "features": { 48 "dind": { "privileged": true }, 49 "sign-key": { 50 "mounts": ["/srv/keys/build.rsa:/keys/build.rsa:ro"], 51 "env": { "UNOS_SIGN_KEY": "/keys/build.rsa" } 52 } 53 } 54 } 55 ``` 56 57 ```sh 58 node src/worker/agent.js --config worker.json 59 ``` 60 61 | Key | Default | Description | 62 | ---------------- | -------------------- | ----------------------------------------------- | 63 | `conductor_url` | `http://127.0.0.1:8080` | Base URL of the conductor. | 64 | `name` | hostname | Identifies this machine to the conductor. | 65 | `token` | none | The worker token, inline. | 66 | `token_file` | none | Path to a file holding the token. Preferred. | 67 | `arches` | `[]` | Architectures offered. Empty means no `arch` tasks, only tasks that declare none. | 68 | `features` | `{}` | Capabilities offered, see below. | 69 | `concurrency` | `1` | Tasks run at once. | 70 | `poll_interval` | `5` | Seconds between polls when idle. | 71 | `docker` | `docker` | Runtime CLI. Set to `podman` or `nerdctl`. | 72 | `shell` | `sh` | Shell used to run a task script in its image. | 73 74 There is nothing here about where tasks run or what they are given, which 75 is deliberate: that is a property of the build and is decided by the 76 pipeline or the project, not by the machine that happens to run it. 77 78 Every key can also be set from the environment: `CONDUCTOR_URL`, 79 `CONDUCTOR_WORKER_NAME`, `CONDUCTOR_WORKER_TOKEN`, 80 `CONDUCTOR_WORKER_TOKEN_FILE`, `CONDUCTOR_WORKER_ARCHES`, 81 `CONDUCTOR_WORKER_CONCURRENCY`, `CONDUCTOR_WORKER_DOCKER`, and so on. The 82 environment wins over the file. 83 84 In a container the default hostname is the container id, which changes on 85 every restart. Set `CONDUCTOR_WORKER_NAME` so a machine keeps one name: 86 under Nomad, `CONDUCTOR_WORKER_NAME = "${attr.unique.hostname}"`. 87 88 Features 89 -------- 90 91 A feature is a name this worker advertises, together with whatever local 92 resources a task asking for it should receive. The conductor only ever learns 93 the name. It will not send a task to a worker that does not advertise every 94 feature the task's `requires` lists. 95 96 The names a machine advertises are recorded against its worker token and 97 shown on that worker's page, so you can see what each machine offers without 98 logging into it. A machine that stops claiming is forgotten after a day. 99 100 ```json 101 "features": { 102 "sign-key": { 103 "mounts": ["/srv/keys/build.rsa:/keys/build.rsa:ro"], 104 "env": { "SIGN_KEY": "/keys/build.rsa" }, 105 "devices": [], 106 "privileged": false 107 } 108 } 109 ``` 110 111 A pipeline then asks for it by name: 112 113 ```yaml 114 package: 115 requires: [sign-key] 116 script: ['./sign.sh'] 117 ``` 118 119 This is how a signing key reaches a build without the conductor ever holding 120 it. The key stays on the machine that owns it, and only tasks that explicitly 121 ask for it are ever scheduled there. 122 123 `privileged: true` is what a docker in docker service needs. Grant it only 124 to features you intend to be privileged, since it removes the isolation 125 between the task and the host. 126 127 What a task gets 128 --------------- 129 130 Each task runs in its own container, on its own network: 131 132 - the repository tree at the task's commit, unpacked into the working 133 directory, which is `/work` unless the pipeline or the project says 134 otherwise 135 - that directory as the working directory 136 - the task's `env`, plus `ARCH` and `MATRIX_*` for a fanned out task 137 - `CONDUCTOR_PROJECT`, `CONDUCTOR_JOB_ID`, `CONDUCTOR_JOB_NUMBER`, 138 `CONDUCTOR_TASK`, `CONDUCTOR_TASK_ID`, `CONDUCTOR_SHA`, `CONDUCTOR_REF` 139 and `CONDUCTOR_ATTEMPT`, which name the project and job the worker 140 itself is never told about: they are opaque strings it copies in 141 - any `services`, reachable by their alias on the task network 142 - mounts and environment from the features it requires 143 144 Output is streamed to the conductor as it happens. Artifacts are read 145 back out of the container when the task finishes, and uploaded. 146 147 Nothing else is shared. The worker mounts no directory of its own into a 148 task: the tree goes in over the docker API and the artifacts come back the 149 same way. That is why a worker needs no volumes, no matching paths 150 between itself and the host, and no storage at all. 151 152 Operational notes 153 ----------------- 154 155 **Cleanup.** Containers and networks are removed whatever the outcome. 156 There is no workspace to clean up, and so no trouble with a task having 157 written files as root that the worker then cannot delete: everything the 158 task wrote lived in its container and went with it. 159 160 **Shutdown.** On SIGINT or SIGTERM the worker stops polling and lets running 161 tasks finish. A second signal exits immediately, and the conductor will 162 eventually reap whatever was left running. 163 164 **Failures.** A worker that stops reporting has its tasks requeued or failed 165 by the conductor after `scheduler.heartbeat_timeout`. Losing a worker never 166 strands a job. 167 168 **Log loss.** If the conductor cannot be reached while a task runs, log output 169 for that window is dropped rather than failing the task. The task's outcome is 170 still reported. 171 172 Trust 173 ----- 174 175 A worker is trusted with the source of the commits it builds and with 176 whatever its features grant it. It is not trusted with anything else: it 177 holds no repository credentials, it cannot read other refs, it cannot reach 178 another project's tasks, and it cannot write to storage except by uploading 179 artifacts for a task it currently holds. 180 181 Running a task means running code from the repository. Only accept work from 182 a conductor whose projects you are willing to execute, and keep features 183 narrow.