README.md (8279B)
1 conductor 2 ========= 3 4 A stand-alone CI system. Pipelines live in the repository, tasks run in 5 containers, and workers pull work rather than being pushed to. 6 7 Summary 8 ------- 9 10 The conductor accepts a push notification, reads `.conductor.yml` at the 11 pushed commit, and turns it into a graph of tasks. Workers poll for work, 12 advertising the architectures and features they can offer. Any task whose 13 dependencies have succeeded is eligible, so a wide graph runs wide. 14 15 Workers need no repository credentials: the conductor serves the source tree 16 for the exact commit a task was handed. A worker on someone else's hardware 17 can therefore build for you without being able to read anything else. 18 19 Nothing has to be set up before it will run. With no configuration it uses 20 sqlite on disk, stores artifacts on the local filesystem, and manages its own 21 user accounts. Point it at a database url, an S3 bucket or an OIDC discovery 22 url and it uses those instead. 23 24 Status 25 ------ 26 27 Under construction. Working today: 28 29 - configuration, with sqlite, mysql/tidb and postgres backends 30 - local and S3 compatible object storage 31 - pipeline parsing, matrix and architecture expansion, dependency graphs 32 - triggers, job scheduling, the worker API, live logs, artifacts, reaping 33 - the worker: containerised tasks, services, features, caches, cleanup 34 - accounts, OIDC, project ownership, visibility and project variables 35 - the web interface, server rendered with htmx 36 - container images for amd64, arm64 and riscv64, published on release 37 38 Installation 39 ------------ 40 41 With docker: 42 43 ```sh 44 docker compose -f deploy/docker-compose.yml up -d conductor 45 ``` 46 47 Images are published as `finwo/conductor` and `finwo/conductor-worker` 48 for amd64, arm64 and riscv64. 49 50 From source, requiring node 24 or newer and no native modules: 51 52 ```sh 53 npm install 54 cp conductor.example.yaml conductor.yaml 55 npm start 56 ``` 57 58 On first start an administrator account is created and its generated 59 password written to the log once. Open the port and sign in. 60 61 `mysql2` and `pg` are optional and only needed when `database.url` points at 62 one of those servers. 63 64 Configuration 65 ------------- 66 67 Every setting has a default, so an empty config file is valid. Values are 68 read from defaults, then `conductor.yaml`, then the environment. See 69 `conductor.example.yaml` for the annotated list. 70 71 Three capabilities switch on when configured and fall back when not: 72 73 | Setting | Configured | Not configured | 74 | ----------------------- | -------------------- | --------------------- | 75 | `database.url` | mysql/tidb, postgres | sqlite at a file path | 76 | `storage.s3.bucket` | S3 compatible store | local filesystem | 77 | `auth.oidc.discovery_url` | OIDC | built-in accounts | 78 79 With OIDC the provider owns identity. Point `auth.oidc.discovery_url` at the 80 provider's OpenID configuration document and set `client_id` (and 81 `client_secret` for a confidential client); the issuer and every endpoint 82 come from that document. Register `<server.public_url>/oidc/callback` as the 83 redirect uri. An account is created locally the first time someone signs in, 84 keyed on issuer and subject, so they can own projects and workers. The role 85 in the token wins on every request, and disabling the local account locks 86 them out regardless of what the provider says. 87 88 Usage 89 ----- 90 91 Register a project and mint a worker token: 92 93 ```sh 94 node src/admin-cli.js project:add demo https://git.example.com/demo.git 95 node src/admin-cli.js token:add builder-1 96 ``` 97 98 Both print a secret once. Put the trigger secret in the git hook and the 99 worker token in the worker configuration. 100 101 Install `hooks/post-receive` into the bare repository: 102 103 ```sh 104 cp hooks/post-receive /srv/git/demo.git/hooks/ 105 chmod +x /srv/git/demo.git/hooks/post-receive 106 ``` 107 108 and set `CONDUCTOR_URL`, `CONDUCTOR_PROJECT` and `CONDUCTOR_SECRET` for it. 109 GitHub, Gitea and GitLab webhooks are accepted at the same endpoint, so a 110 forge can drive it instead. 111 112 Then run a worker, on this machine or any other: 113 114 ```sh 115 node src/worker/agent.js --config worker.json 116 ``` 117 118 See [docs/worker.md](docs/worker.md), and `examples/worker.json`. 119 120 Who sees what 121 ------------- 122 123 Everything is private by default. A job is visible to an anonymous visitor 124 only when the project, or the pipeline at that commit, says it is public: 125 126 ```yaml 127 version: 1 128 visibility: public 129 ``` 130 131 Because it is recorded per job, making a repository private stops exposing 132 future jobs without rewriting the history of past ones. 133 134 Users register their own projects and their own workers. A worker someone 135 registers is only ever offered tasks from that person's projects, which is 136 what makes it safe to accept build capacity from people you do not 137 otherwise trust. A worker created by an administrator with no owner is 138 shared, and runs anything. 139 140 Administrators see and manage everything, and manage accounts. Deleting an 141 account deletes the projects and workers it owned, along with their job 142 history: leaving them behind would turn a personal worker into a shared one. 143 144 Pipelines 145 --------- 146 147 ```yaml 148 version: 1 149 150 defaults: 151 image: debian:bookworm-slim 152 timeout: 30m 153 154 tasks: 155 test: 156 script: 157 - make test 158 159 build: 160 arch: [x86_64, aarch64] 161 script: 162 - ./build.sh $ARCH 163 artifacts: 164 paths: [dist/**] 165 166 package: 167 needs: [build] 168 arch: [x86_64, aarch64] 169 matrix: 170 pkg: [musl, busybox] 171 requires: [sign-key] 172 script: 173 - ./package.sh $MATRIX_PKG $ARCH 174 175 publish: 176 needs: [package, test] 177 script: 178 - ./publish.sh 179 ``` 180 181 `arch` and `matrix` fan a task out into one task per combination. A dependency 182 that shares a dimension is matched on it, so the aarch64 package waits for 183 the aarch64 build rather than for every build, while `publish`, which has no 184 architecture of its own, waits for all of them. 185 186 `requires` names features a worker must offer. A worker holding a signing key 187 advertises `sign-key`, and only tasks asking for it are sent there. The key 188 stays on the worker and is never known to the conductor. 189 190 See [docs/pipeline.md](docs/pipeline.md) for the full reference, 191 [docs/worker.md](docs/worker.md) for running a worker, 192 [docs/deployment.md](docs/deployment.md) for deploying, and 193 [docs/api.md](docs/api.md) for the HTTP API. 194 195 Architecture 196 ------------ 197 198 ``` 199 git push -> post-receive -> conductor -> task graph in the database 200 | 201 worker polls 202 | 203 source tarball, container, log stream, artifacts 204 ``` 205 206 - `src/conductor` triggers, scheduling, the worker API, the interface 207 - `src/worker` the agent that runs tasks, with no npm dependencies 208 - `src/lib` configuration, database, storage, pipelines, git 209 210 Queries are written once against all three databases using named `{key}` 211 markers, which each driver compiles to its own placeholder syntax. 212 Timestamps are epoch milliseconds everywhere. 213 214 Testing 215 ------- 216 217 ```sh 218 npm test 219 ``` 220 221 That is the whole setup. With docker available the suite starts what it 222 needs and cleans up after itself: 223 224 | Dependency | Used for | 225 | ------------------- | -------------------------------------------- | 226 | `alpine` | containers for the worker to actually run tasks in | 227 | MinIO | object storage, artifacts and archived logs | 228 | mock-oauth2-server | a real OIDC provider to authenticate against | 229 230 Nothing is stubbed, because the failures worth catching live in the parts a 231 stub would replace. Hand written SigV4 signing is only meaningfully tested 232 by a server that rejects a bad signature, and the JWKS path was wrong 233 against every provider except the one it was written for. 234 235 Without docker those tests skip and the rest still run: 236 237 ```sh 238 CONDUCTOR_TEST_NO_DOCKER=1 npm test 239 ``` 240 241 To test against a different object store, such as Garage or AWS, point the 242 same tests at it instead of starting MinIO: 243 244 ```sh 245 docker run -d -p 9000:9000 -e MINIO_ROOT_USER=testkey \ 246 -e MINIO_ROOT_PASSWORD=testsecret123 quay.io/minio/minio server /data 247 248 CONDUCTOR_TEST_S3_ENDPOINT=http://127.0.0.1:9000 \ 249 CONDUCTOR_TEST_S3_KEY=testkey \ 250 CONDUCTOR_TEST_S3_SECRET=testsecret123 npm test 251 ``` 252 253 LICENSE 254 ------- 255 256 See [LICENSE.md](LICENSE.md).