deployment.md (11771B)
1 Deployment 2 ========== 3 4 One service, one image. The conductor accepts triggers, schedules jobs, 5 serves the worker API and renders the interface. Workers are separate and 6 may live anywhere, including on machines you do not administer. 7 8 Quick start 9 ----------- 10 11 ```sh 12 docker compose -f deploy/docker-compose.yml up -d 13 ``` 14 15 That pulls the published images and brings up a conductor on port 8080 16 with sqlite on a volume, and one worker. The compose file needs a worker token, so the first job is 17 two steps: 18 19 ```sh 20 # Start the conductor on its own. 21 docker compose -f deploy/docker-compose.yml up -d conductor 22 23 # Mint a token, then bring the worker up with it. 24 docker compose -f deploy/docker-compose.yml exec conductor \ 25 node src/admin-cli.js token:add compose-worker 26 27 CONDUCTOR_WORKER_TOKEN=... docker compose -f deploy/docker-compose.yml up -d worker 28 ``` 29 30 The administrator password is written to the conductor log once on first 31 start. Set `CONDUCTOR_ADMIN_PASSWORD` to choose it instead. 32 33 To check the whole thing before trusting it with anything: 34 35 ```sh 36 deploy/smoke.sh 37 ``` 38 39 That builds both images, starts them, runs a real pipeline and verifies the 40 log and artifact came back. 41 42 Images 43 ------ 44 45 | Image | Contents | 46 | ------------------------- | ------------------------------------------- | 47 | `finwo/conductor` | the conductor: node, git, the application | 48 | `finwo/conductor-worker` | the worker: node, git, tar, the docker CLI | 49 50 Published for `linux/amd64`, `linux/arm64` and `linux/riscv64`. Tags: 51 `latest` follows releases, `1.2.3` and `1.2` pin a version, and a twelve 52 character commit id pins an exact build from main. 53 54 Both are built on Alpine rather than the official node image, which is 55 only published for amd64, arm64 and ppc64le. Alpine 3.23 carries node 24 56 on every architecture here, and a docker client too, so the worker no 57 longer fetches a static binary that has no riscv64 build at all. 58 59 To build them yourself: 60 61 ```sh 62 docker build -f deploy/Dockerfile -t finwo/conductor . 63 docker build -f deploy/Dockerfile.worker -t finwo/conductor-worker . 64 ``` 65 66 The worker image installs `yaml` so its configuration file may be YAML; the 67 worker source itself has no npm dependencies at all. 68 69 State 70 ----- 71 72 Everything the conductor keeps lives in `/data`: 73 74 ``` 75 /data/conductor.db sqlite, unless database.url points elsewhere 76 /data/mirrors one bare mirror per project 77 /data/logs the live log spool 78 /data/storage artifacts and archived logs, unless S3 is configured 79 ``` 80 81 Mount a volume there. Losing it loses job history, and the mirrors and 82 spool will be rebuilt. 83 84 Configuration 85 ------------- 86 87 Every setting has a default and can come from the environment, so the image 88 needs no config file. See `conductor.example.yaml` for the annotated list. 89 90 Worth setting in production: 91 92 | Variable | Why | 93 | -------------------------- | ------------------------------------------------ | 94 | `CONDUCTOR_PUBLIC_URL` | Workers get absolute callback URLs built from it. | 95 | `CONDUCTOR_SESSION_SECRET` | Otherwise sessions end at every restart. | 96 | `CONDUCTOR_SECRET_KEY` | Otherwise stored secrets are kept in the clear. | 97 | `CONDUCTOR_ADMIN_PASSWORD` | Otherwise one is generated and logged once. | 98 99 Generate the two secrets with `openssl rand -hex 32`. 100 101 ### Postgres or MySQL 102 103 ```sh 104 CONDUCTOR_DATABASE_URL=postgres://conductor:secret@postgres:5432/conductor \ 105 docker compose --profile postgres -f deploy/docker-compose.yml up -d 106 ``` 107 108 The schema is created on start. `npm run migrate` applies it separately 109 when a deployment wants that as its own reviewable step. 110 111 ### Object storage 112 113 ```sh 114 CONDUCTOR_S3_ENDPOINT=http://minio:9000 \ 115 CONDUCTOR_S3_BUCKET=conductor \ 116 CONDUCTOR_S3_ACCESS_KEY_ID=... \ 117 CONDUCTOR_S3_SECRET_ACCESS_KEY=... \ 118 docker compose --profile s3 -f deploy/docker-compose.yml up -d 119 ``` 120 121 Artifacts and finished logs then go to the bucket instead of the volume, 122 and downloads are handed to the store with a presigned redirect rather than 123 being proxied. The bucket must already exist. 124 125 Workers 126 ------- 127 128 A worker needs the docker socket, and nothing else: 129 130 ```sh 131 docker run -d --restart unless-stopped \ 132 -e CONDUCTOR_URL=https://ci.example.com \ 133 -e CONDUCTOR_WORKER_TOKEN=... \ 134 -v /var/run/docker.sock:/var/run/docker.sock \ 135 finwo/conductor-worker 136 ``` 137 138 It keeps no state and shares no directory with the tasks it runs. A task's 139 tree is unpacked into the task's own container over the socket, and its 140 artifacts are read back the same way, so there is no workspace to mount, 141 no path that has to mean the same thing on both sides, and nothing to 142 persist between restarts. 143 144 Feature mounts are the one exception, and they are deliberate: those name 145 host paths an operator chose, such as a signing key, and are declared in 146 the worker's configuration file. See [worker.md](worker.md). 147 148 To contribute capacity to somebody else's conductor, use 149 `deploy/worker/docker-compose.yml`. 150 151 Behind a reverse proxy 152 ---------------------- 153 154 ``` 155 server { 156 server_name ci.example.com; 157 location / { 158 proxy_pass http://127.0.0.1:8080; 159 proxy_set_header Host $host; 160 proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for; 161 proxy_set_header X-Forwarded-Proto $scheme; 162 # Artifacts can be large, and a build can be quiet for a long time. 163 client_max_body_size 0; 164 proxy_read_timeout 3600s; 165 proxy_request_buffering off; 166 } 167 } 168 ``` 169 170 `proxy_request_buffering off` matters: with it on, nginx spools an entire 171 artifact to its own disk before the conductor sees a byte, which for 172 multi gigabyte build output is both slow and surprising. 173 174 Set `CONDUCTOR_PUBLIC_URL=https://ci.example.com` to match, or workers will 175 be handed callback URLs that do not work. 176 177 Retention 178 --------- 179 180 Nothing is deleted unless this says so, and logs are what fills a disk: 181 written for every task, read for almost none. The defaults keep artifacts 182 for 30 days or the last 10 jobs, and logs for 14 days. 183 184 ```yaml 185 retention: 186 artifact_keep_jobs: 10 187 artifact_keep_days: 30 188 log_keep_days: 14 189 sweep_interval: 3600 190 batch: 500 191 ``` 192 193 A project overrides any of these from its settings page. 194 195 Three values, three meanings, and the difference matters: 196 197 | Value | Meaning | 198 | ------ | ---------------------------------------- | 199 | `null` | follow the server default | 200 | `0` | keep forever | 201 | `n` | keep that many jobs, or that many days | 202 203 The settings page shows what a `null` resolves to, since the server defaults 204 are not otherwise visible there. 205 206 ### How the artifact rules combine 207 208 An artifact is kept if **either** rule wants it. The last 209 `artifact_keep_jobs` jobs keep their artifacts however old they are, and 210 anything younger than `artifact_keep_days` is kept however many jobs have 211 followed. Setting both is therefore more generous than setting one, not 212 less. 213 214 The most recent **successful** job is kept regardless, so a project that 215 has not built in months still has something to download. Only the latest 216 one: older successes are not protected. 217 218 A task that sets `artifacts.expire` in its pipeline overrides all of the 219 above with an exact deadline, including the protection for the last good 220 job. That is deliberate, and is how a bulky intermediate avoids becoming 221 immortal by being green. 222 223 Logs go by age alone. Sweeping a log deletes the bytes and leaves the task, 224 so a job stays explainable after its output is gone, and the interface can 225 tell an expired log apart from one that was never written. 226 227 ### What a first sweep will do 228 229 Upgrading applies these defaults to everything already stored. On an 230 installation that has been running for a while that first sweep will 231 delete a great deal, so set the values you want before starting the new 232 version, or set them to `0` and decide later: 233 234 ```sh 235 CONDUCTOR_RETENTION_ARTIFACT_DAYS=0 \ 236 CONDUCTOR_RETENTION_LOG_DAYS=0 \ 237 docker compose -f deploy/docker-compose.yml up -d 238 ``` 239 240 A sweep deletes at most `batch` artifacts and `batch` logs per project per 241 pass, so a large backlog drains over several passes rather than in one 242 long transaction. Objects are deleted before rows: an object left behind 243 wastes space, whereas a row left behind is a download that fails. 244 245 Backups 246 ------- 247 248 With sqlite and local storage, the `/data` volume is the whole system. With 249 postgres and S3, back up the database and the bucket; the mirrors and the 250 log spool are caches and rebuild themselves. 251 252 Secrets are encrypted with `CONDUCTOR_SECRET_KEY`. A backup restored 253 without that key leaves trigger secrets and project variables unreadable, 254 so keep it somewhere other than next to the backup. 255 256 Upgrading 257 --------- 258 259 Migrations run on start and are forward only. They are recorded with a 260 checksum, so an edited migration is refused rather than applied twice. 261 Take a backup first: there is no automatic downgrade. 262 263 ```sh 264 docker compose -f deploy/docker-compose.yml pull 265 docker compose -f deploy/docker-compose.yml up -d 266 ``` 267 268 Pin a version rather than tracking `latest` if an unattended restart 269 picking up a new release would be unwelcome: 270 271 ```sh 272 CONDUCTOR_IMAGE=finwo/conductor:1.2.3 \ 273 CONDUCTOR_WORKER_IMAGE=finwo/conductor-worker:1.2.3 \ 274 docker compose -f deploy/docker-compose.yml up -d 275 ``` 276 277 Workers can be upgraded independently and in any order. The worker API is 278 the boundary between them, and a worker that goes away mid-task has that task 279 requeued after `scheduler.heartbeat_timeout`. 280 281 Publishing 282 ---------- 283 284 The images are built and pushed by the project's own pipeline, from 285 `deploy/publish.sh`. It runs only on `main` and on `v*` tags, enforced by 286 an `only:` rule in `.conductor.yml` rather than by a check inside the 287 script. The script itself handles any branch, so publishing a branch is a 288 matter of widening that rule. 289 290 | Ref | Tags pushed | 291 | ------------------- | ------------------ | 292 | `refs/heads/main` | `main`, `latest` | 293 | `refs/heads/topic` | `topic` | 294 | `refs/tags/v1.2.3` | `1.2.3`, `1.2` | 295 296 A branch publishes its own name, with any aliases listed for it in 297 `branch_aliases`. Only `main` has one, `latest`, and nothing else moves 298 it, so `latest` is always whatever main is. Branch names are rewritten to 299 suit a docker tag, so `feature/x` publishes as `feature-x`. 300 301 A tag that is not a plain version, `v1.2.3-rc1` for instance, is refused 302 rather than guessed at. 303 304 Pushing needs `REGISTRY_USERNAME` and `REGISTRY_TOKEN` as project 305 variables. They are masked in task logs, and passed to `docker login` on 306 standard input rather than on the command line so they stay out of the 307 process list. The task logs out again on the way out, whether or not it 308 succeeded, so nothing is left behind on a shared worker. 309 310 The foreign architectures are emulated with QEMU, registered per job with 311 `tonistiigi/binfmt`. Giving the pool a worker of that architecture makes 312 it native instead, since a worker declares its own architectures when it 313 polls. 314 315 Emulation needs `binfmt_misc` mounted on the host running the docker 316 daemon, which is not the default everywhere: 317 318 ```sh 319 sudo mount -t binfmt_misc binfmt_misc /proc/sys/fs/binfmt_misc 320 ``` 321 322 Without it the registration step still reports success, because the 323 handlers are written into the container's own mount namespace and go away 324 with it. The build then fails much later with `exec format error`. 325 `publish.sh` checks that the builder really offers every platform before 326 building anything, and says this if it does not. 327 328 To see what a release would push without pushing it: 329 330 ```sh 331 CONDUCTOR_REF=refs/tags/v1.2.3 CONDUCTOR_SHA=$(git rev-parse HEAD) \ 332 DRY_RUN=1 deploy/publish.sh 333 ```