deployment.md (13470B)
1 Deployment 2 ========== 3 4 One service, one image. The conductor accepts triggers, schedules jobs, 5 serves the API and hosts the application that drives it. Workers are 6 separate and may live anywhere, including on machines you do not 7 administer. 8 9 Quick start 10 ----------- 11 12 ```sh 13 docker compose -f deploy/docker-compose.yml up -d 14 ``` 15 16 That pulls the published images and brings up a conductor on port 8080 17 with sqlite on a volume, and one worker. The compose file needs a worker token, so the first job is 18 two steps: 19 20 ```sh 21 # Start the conductor on its own. 22 docker compose -f deploy/docker-compose.yml up -d conductor 23 24 # Mint a token, then bring the worker up with it. 25 docker compose -f deploy/docker-compose.yml exec conductor \ 26 node src/admin-cli.js token:add compose-worker 27 28 CONDUCTOR_WORKER_TOKEN=... docker compose -f deploy/docker-compose.yml up -d worker 29 ``` 30 31 The administrator password is written to the conductor log once on first 32 start. Set `CONDUCTOR_ADMIN_PASSWORD` to choose it instead. 33 34 To check the whole thing before trusting it with anything: 35 36 ```sh 37 deploy/smoke.sh 38 ``` 39 40 That builds both images, starts them, runs a real pipeline and verifies the 41 log and artifact came back. 42 43 Images 44 ------ 45 46 | Image | Contents | 47 | ------------------------- | ------------------------------------------- | 48 | `finwo/conductor` | the conductor: node, git, the application | 49 | `finwo/conductor-worker` | the worker: node, git, tar, the docker CLI | 50 51 Published for `linux/amd64`, `linux/arm64` and `linux/riscv64`. Tags: 52 `latest` follows releases, `1.2.3` and `1.2` pin a version, and a twelve 53 character commit id pins an exact build from main. 54 55 Both are built on Alpine rather than the official node image, which is 56 only published for amd64, arm64 and ppc64le. Alpine 3.23 carries node 24 57 on every architecture here, and a docker client too, so the worker no 58 longer fetches a static binary that has no riscv64 build at all. 59 60 To build them yourself: 61 62 ```sh 63 docker build -f deploy/Dockerfile -t finwo/conductor . 64 docker build -f deploy/Dockerfile.worker -t finwo/conductor-worker . 65 ``` 66 67 The worker image installs `yaml` so its configuration file may be YAML; the 68 worker source itself has no npm dependencies at all. 69 70 State 71 ----- 72 73 Everything the conductor keeps lives in `/data`: 74 75 ``` 76 /data/conductor.db sqlite, unless database.url points elsewhere 77 /data/mirrors one bare mirror per project 78 /data/logs the live log spool 79 /data/storage artifacts and archived logs, unless S3 is configured 80 ``` 81 82 Mount a volume there. Losing it loses job history, and the mirrors and 83 spool will be rebuilt. 84 85 Configuration 86 ------------- 87 88 Every setting has a default and can come from the environment, so the image 89 needs no config file. See `conductor.example.yaml` for the annotated list. 90 91 Worth setting in production: 92 93 | Variable | Why | 94 | -------------------------- | ------------------------------------------------ | 95 | `CONDUCTOR_PUBLIC_URL` | Workers get absolute callback URLs built from it. | 96 | `CONDUCTOR_SESSION_SECRET` | Otherwise access tokens stop verifying at every restart. | 97 | `CONDUCTOR_SECRET_KEY` | Otherwise stored secrets are kept in the clear. | 98 | `CONDUCTOR_ADMIN_PASSWORD` | Otherwise one is generated and logged once. | 99 100 Generate the two secrets with `openssl rand -hex 32`. 101 102 ### Postgres or MySQL 103 104 ```sh 105 CONDUCTOR_DATABASE_URL=postgres://conductor:secret@postgres:5432/conductor \ 106 docker compose --profile postgres -f deploy/docker-compose.yml up -d 107 ``` 108 109 The schema is created on start. `npm run migrate` applies it separately 110 when a deployment wants that as its own reviewable step. 111 112 ### Object storage 113 114 ```sh 115 CONDUCTOR_S3_ENDPOINT=http://minio:9000 \ 116 CONDUCTOR_S3_BUCKET=conductor \ 117 CONDUCTOR_S3_ACCESS_KEY_ID=... \ 118 CONDUCTOR_S3_SECRET_ACCESS_KEY=... \ 119 docker compose --profile s3 -f deploy/docker-compose.yml up -d 120 ``` 121 122 Artifacts and finished logs then go to the bucket instead of the volume, 123 and downloads are handed to the store with a presigned redirect rather than 124 being proxied. The bucket must already exist. 125 126 Workers 127 ------- 128 129 A worker needs the docker socket, and nothing else: 130 131 ```sh 132 docker run -d --restart unless-stopped \ 133 -e CONDUCTOR_URL=https://ci.example.com \ 134 -e CONDUCTOR_WORKER_TOKEN=... \ 135 -v /var/run/docker.sock:/var/run/docker.sock \ 136 finwo/conductor-worker 137 ``` 138 139 It keeps no state and shares no directory with the tasks it runs. A task's 140 tree is unpacked into the task's own container over the socket, and its 141 artifacts are read back the same way, so there is no workspace to mount, 142 no path that has to mean the same thing on both sides, and nothing to 143 persist between restarts. 144 145 Feature mounts are the one exception, and they are deliberate: those name 146 host paths an operator chose, such as a signing key, and are declared in 147 the worker's configuration file. See [worker.md](worker.md). 148 149 To contribute capacity to somebody else's conductor, use 150 `deploy/worker/docker-compose.yml`. 151 152 Behind a reverse proxy 153 ---------------------- 154 155 ``` 156 server { 157 server_name ci.example.com; 158 location / { 159 proxy_pass http://127.0.0.1:8080; 160 proxy_set_header Host $host; 161 proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for; 162 proxy_set_header X-Forwarded-Proto $scheme; 163 # Artifacts can be large, and a build can be quiet for a long time. 164 client_max_body_size 0; 165 proxy_read_timeout 3600s; 166 proxy_request_buffering off; 167 } 168 } 169 ``` 170 171 `proxy_request_buffering off` matters: with it on, nginx spools an entire 172 artifact to its own disk before the conductor sees a byte, which for 173 multi gigabyte build output is both slow and surprising. 174 175 Set `CONDUCTOR_PUBLIC_URL=https://ci.example.com` to match, or workers will 176 be handed callback URLs that do not work. It is also the issuer the 177 built-in provider names in its tokens, and the default redirect uri the 178 application is allowed to come back to. 179 180 Nothing special is needed for the application's own routing. It routes on 181 the fragment, so a deep link is `https://ci.example.com/#!/jobs/<id>` and 182 the path the proxy sees is always the root. There is no history fallback 183 to configure. 184 185 ### Serving the application from somewhere else 186 187 The bundle in `web/dist` is static and can be served by anything. If it is 188 not on the conductor's own origin, two things have to be said: 189 190 ```sh 191 CONDUCTOR_LOCAL_REDIRECT_URIS=https://ci.example.com,https://app.example.com 192 ``` 193 194 so the built-in provider will send a browser back there, and the 195 application still needs to reach `/.well-known/openid-configuration` and 196 `/api/v1` on the conductor. CORS is already open, but the browser will 197 refuse the cross-origin fetch unless the Content-Security-Policy on 198 whatever serves the bundle allows it. 199 200 Hosting it on the conductor itself avoids all of that, which is why that is 201 the default. 202 203 Retention 204 --------- 205 206 Nothing is deleted unless this says so, and logs are what fills a disk: 207 written for every task, read for almost none. The defaults keep artifacts 208 for 30 days or the last 10 jobs, and logs for 14 days. 209 210 ```yaml 211 retention: 212 artifact_keep_jobs: 10 213 artifact_keep_days: 30 214 log_keep_days: 14 215 sweep_interval: 3600 216 batch: 500 217 ``` 218 219 A project overrides any of these from its settings page. 220 221 Three values, three meanings, and the difference matters: 222 223 | Value | Meaning | 224 | ------ | ---------------------------------------- | 225 | `null` | follow the server default | 226 | `0` | keep forever | 227 | `n` | keep that many jobs, or that many days | 228 229 The settings page shows what a `null` resolves to, since the server defaults 230 are not otherwise visible there. 231 232 ### How the artifact rules combine 233 234 An artifact is kept if **either** rule wants it. The last 235 `artifact_keep_jobs` jobs keep their artifacts however old they are, and 236 anything younger than `artifact_keep_days` is kept however many jobs have 237 followed. Setting both is therefore more generous than setting one, not 238 less. 239 240 The most recent **successful** job is kept regardless, so a project that 241 has not built in months still has something to download. Only the latest 242 one: older successes are not protected. 243 244 A task that sets `artifacts.expire` in its pipeline overrides all of the 245 above with an exact deadline, including the protection for the last good 246 job. That is deliberate, and is how a bulky intermediate avoids becoming 247 immortal by being green. 248 249 Logs go by age alone. Sweeping a log deletes the bytes and leaves the task, 250 so a job stays explainable after its output is gone, and a reader is told 251 the log expired rather than that it was empty. 252 253 The same sweep forgets machines that have stopped claiming, after 254 `retention.worker_agent_keep_hours` (default 24, `0` keeps them for ever). 255 A machine that comes back reappears on its next claim. 256 257 ### What a first sweep will do 258 259 Upgrading applies these defaults to everything already stored. On an 260 installation that has been running for a while that first sweep will 261 delete a great deal, so set the values you want before starting the new 262 version, or set them to `0` and decide later: 263 264 ```sh 265 CONDUCTOR_RETENTION_ARTIFACT_DAYS=0 \ 266 CONDUCTOR_RETENTION_LOG_DAYS=0 \ 267 docker compose -f deploy/docker-compose.yml up -d 268 ``` 269 270 A sweep deletes at most `batch` artifacts and `batch` logs per project per 271 pass, so a large backlog drains over several passes rather than in one 272 long transaction. Objects are deleted before rows: an object left behind 273 wastes space, whereas a row left behind is a download that fails. 274 275 Backups 276 ------- 277 278 With sqlite and local storage, the `/data` volume is the whole system. With 279 postgres and S3, back up the database and the bucket; the mirrors and the 280 log spool are caches and rebuild themselves. 281 282 Secrets are encrypted with `CONDUCTOR_SECRET_KEY`. A backup restored 283 without that key leaves trigger secrets and project variables unreadable, 284 so keep it somewhere other than next to the backup. 285 286 Upgrading 287 --------- 288 289 Migrations run on start and are forward only. They are recorded with a 290 checksum, so an edited migration is refused rather than applied twice. 291 Take a backup first: there is no automatic downgrade. 292 293 ```sh 294 docker compose -f deploy/docker-compose.yml pull 295 docker compose -f deploy/docker-compose.yml up -d 296 ``` 297 298 Pin a version rather than tracking `latest` if an unattended restart 299 picking up a new release would be unwelcome: 300 301 ```sh 302 CONDUCTOR_IMAGE=finwo/conductor:1.2.3 \ 303 CONDUCTOR_WORKER_IMAGE=finwo/conductor-worker:1.2.3 \ 304 docker compose -f deploy/docker-compose.yml up -d 305 ``` 306 307 Workers are usually upgraded independently and in any order. The worker API 308 is the boundary between them, and a worker that goes away mid-task has that 309 task requeued after `scheduler.heartbeat_timeout`. 310 311 Two releases break that, and need the conductor and every worker restarted 312 together: **0.5.0**, which put every API response in an envelope, and 313 **0.3.0**, which changed what a worker claims. Across either boundary a 314 worker of the older version cannot read what the conductor sends, so it 315 claims nothing until it is upgraded. Pull on every host first and restart 316 afterwards, so the window is seconds rather than the length of a download. 317 318 Publishing 319 ---------- 320 321 The images are built and pushed by the project's own pipeline, from 322 `deploy/publish.sh`. It runs only on `main` and on `v*` tags, enforced by 323 an `only:` rule in `.conductor.yml` rather than by a check inside the 324 script. The script itself handles any branch, so publishing a branch is a 325 matter of widening that rule. 326 327 | Ref | Tags pushed | 328 | ------------------- | ------------------ | 329 | `refs/heads/main` | `main`, `latest` | 330 | `refs/heads/topic` | `topic` | 331 | `refs/tags/v1.2.3` | `1.2.3`, `1.2` | 332 333 A branch publishes its own name, with any aliases listed for it in 334 `branch_aliases`. Only `main` has one, `latest`, and nothing else moves 335 it, so `latest` is always whatever main is. Branch names are rewritten to 336 suit a docker tag, so `feature/x` publishes as `feature-x`. 337 338 A tag that is not a plain version, `v1.2.3-rc1` for instance, is refused 339 rather than guessed at. 340 341 Pushing needs `REGISTRY_USERNAME` and `REGISTRY_TOKEN` as project 342 variables. They are masked in task logs, and passed to `docker login` on 343 standard input rather than on the command line so they stay out of the 344 process list. The task logs out again on the way out, whether or not it 345 succeeded, so nothing is left behind on a shared worker. 346 347 The foreign architectures are emulated with QEMU, registered per job with 348 `tonistiigi/binfmt`. Giving the pool a worker of that architecture makes 349 it native instead, since a worker declares its own architectures when it 350 polls. 351 352 Emulation needs `binfmt_misc` mounted on the host running the docker 353 daemon, which is not the default everywhere: 354 355 ```sh 356 sudo mount -t binfmt_misc binfmt_misc /proc/sys/fs/binfmt_misc 357 ``` 358 359 Without it the registration step still reports success, because the 360 handlers are written into the container's own mount namespace and go away 361 with it. The build then fails much later with `exec format error`. 362 `publish.sh` checks that the builder really offers every platform before 363 building anything, and says this if it does not. 364 365 To see what a release would push without pushing it: 366 367 ```sh 368 CONDUCTOR_REF=refs/tags/v1.2.3 CONDUCTOR_SHA=$(git rev-parse HEAD) \ 369 DRY_RUN=1 deploy/publish.sh 370 ```