conductor

CI task system
git clone git://git.finwo.net/app/conductor
Log | Files | Refs | README | LICENSE

commit f55d8d39bc2f992264e889655daec9833eae68d8
parent e904c928c3207919a3a43bfbc97709be0437e834
Author: finwo <finwo@pm.me>
Date:   Sat, 26 Sep 2026 18:00:29 +0200

Document the API, the two providers and the code flow

Diffstat:
M.conductor.yml | 9++++++++-
MREADME.md | 41++++++++++++++++++++++++++++-------------
Mdeploy/hub/conductor-worker.md | 2+-
Mdeploy/hub/conductor.md | 2+-
Mdocs/api.md | 193+++++++++++++++++++++++++++++++++++++++++++++++++++++++++----------------------
Adocs/auth.md | 196+++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
Mdocs/deployment.md | 38++++++++++++++++++++++++++++++++------
Mtest/self-pipeline.test.js | 4++--
8 files changed, 407 insertions(+), 78 deletions(-)

diff --git a/.conductor.yml b/.conductor.yml @@ -22,6 +22,13 @@ tasks: - npm install --no-audit --no-fund - node --test test/ascii.test.js + # The bundle the image ships. Built here too, so a broken frontend is + # caught by itself rather than as a confusing Dockerfile failure. + build: + script: + - npm install --no-audit --no-fund + - npm run build + # Everything that runs without a docker socket. test: script: @@ -44,7 +51,7 @@ tasks: # Host architecture only, since this runs on every branch and the point # is to catch a broken Dockerfile early rather than to ship anything. images: - needs: [style, test] + needs: [style, build, test] requires: [docker] image: docker:28-cli script: diff --git a/README.md b/README.md @@ -32,7 +32,7 @@ Under construction. Working today: - triggers, job scheduling, the worker API, live logs, artifacts, reaping - the worker: containerised tasks, services, features, caches, cleanup - accounts, OIDC, project ownership, visibility and project variables - - the web interface, server rendered with htmx + - a JSON API, and a single page application that is a client of it - container images for amd64, arm64 and riscv64, published on release Installation @@ -74,16 +74,23 @@ Three capabilities switch on when configured and fall back when not: | ----------------------- | -------------------- | --------------------- | | `database.url` | mysql/tidb, postgres | sqlite at a file path | | `storage.s3.bucket` | S3 compatible store | local filesystem | -| `auth.oidc.discovery_url` | OIDC | built-in accounts | +| `auth.oidc.discovery_url` | that provider | the built-in login page | -With OIDC the provider owns identity. Point `auth.oidc.discovery_url` at the -provider's OpenID configuration document and set `client_id` (and -`client_secret` for a confidential client); the issuer and every endpoint -come from that document. Register `<server.public_url>/oidc/callback` as the -redirect uri. An account is created locally the first time someone signs in, -keyed on issuer and subject, so they can own projects and workers. The role -in the token wins on every request, and disabling the local account locks -them out regardless of what the provider says. +Authentication is an authorization code flow either way, and the +application cannot tell the two apart: it reads +`/.well-known/openid-configuration` from this conductor and uses whatever +that names. Only one provider is ever advertised. + +With a provider configured it owns identity. Register conductor as a +**public** client with PKCE; there is no client secret, because the +application redeems the code itself and the conductor never does. An +account is created locally the first time someone signs in, keyed on issuer +and subject, so they can own projects and workers. The role in the token +wins on every request, and disabling the local account locks them out +regardless of what the provider says. + +`docs/auth.md` has the whole of it, including the Keycloak setup and the +audience claim that catches everyone out. Usage ----- @@ -221,9 +228,10 @@ git push -> post-receive -> conductor -> task graph in the database source tarball, container, log stream, artifacts ``` - - `src/conductor` triggers, scheduling, the worker API, the interface + - `src/conductor` triggers, scheduling, the API, the login provider - `src/worker` the agent that runs tasks, with no npm dependencies - `src/lib` configuration, database, storage, pipelines, git + - `web` the application, a mithril client of the API Queries are written once against all three databases using named `{key}` markers, which each driver compiles to its own placeholder syntax. @@ -236,8 +244,15 @@ Testing npm test ``` -That is the whole setup. With docker available the suite starts what it -needs and cleans up after itself: +That is the whole setup. The application is built separately, and the +container image builds it in its own stage: + +```sh +npm run build +``` + +With docker available the suite starts what it needs and cleans up after +itself: | Dependency | Used for | | ------------------- | -------------------------------------------- | diff --git a/deploy/hub/conductor-worker.md b/deploy/hub/conductor-worker.md @@ -32,7 +32,7 @@ Pin both to the same version if they are upgraded separately. ## Quick start A worker needs no inbound connectivity, so this works from behind NAT. -Mint a token in the interface under Workers, or with the admin CLI on the +Mint a token under Workers in the application, or with the admin CLI on the server, then: ```sh diff --git a/deploy/hub/conductor.md b/deploy/hub/conductor.md @@ -1,7 +1,7 @@ # conductor A small CI server. Accepts triggers, schedules jobs, serves the worker -API and renders the interface. One service, one image. +API and hosts the application that drives it. One service, one image. Source and issues: https://github.com/finwo/conductor diff --git a/docs/api.md b/docs/api.md @@ -12,25 +12,30 @@ A worker deals only in tasks. It is never told which project or job a task belongs to, and the task id it receives carries no structure it could read that out of. What it gets is a context and a script. -The conductor exposes five HTTP surfaces, all under `/api/v1`. Everything -else, including signing in and managing projects, workers and users, -happens in the interface. - -| Surface | Authentication | -| --------------------------------------------- | --------------------- | -| `POST /api/v1/projects/:project/trigger` | per project HMAC | -| `POST /api/v1/projects/:project/jobs` | per project HMAC | -| `GET /api/v1/projects/:project/jobs/:job` | HMAC, session, or anonymous for a public job | -| `GET /api/v1/tasks/:task` | session, or anonymous for a public task | -| `GET /api/v1/projects/:project/tasks/:task` | HMAC, session, or anonymous for a public task | -| `POST /api/v1/tasks/claim`, `/tasks/:task/...` | worker token | -| `GET /api/v1/projects/.../artifacts/...` | session, or anonymous for a public job | +Everything the conductor does is under `/api/v1`. There is no second +surface: the application in `web/` is an ordinary client of what follows, +and the only page the conductor renders itself is the built-in login form. + +Three kinds of credential reach it, all in `Authorization: Bearer`: + +| Credential | Obtained by | Reaches | +| ------------------- | ---------------------------------- | --------------------------- | +| access token | the code flow, see `docs/auth.md` | everything a person may do | +| worker token | `POST /api/v1/workers` | the worker protocol only | +| project HMAC | a project's trigger secret | that project's jobs | + +An access token is a JWT and a worker token is 64 hex characters, so the +two never parse as each other. The project HMAC is not a bearer credential +at all: it is `X-Hub-Signature-256` over the exact request body. Note the two readings of `/api/v1/tasks/:task`. Asking after the task itself is a public read; everything below it is the worker protocol and needs a token. They are separate plugins for that reason, so the token requirement cannot accidentally spread to the first or lapse on the rest. +Anonymous is not an error. A public project's jobs and tasks read without +any credential at all. + Responses --------- @@ -137,54 +142,71 @@ Reading a job back ------------------ ``` -GET /api/v1/projects/:project/jobs/:job +GET /api/v1/jobs every job the caller may see +GET /api/v1/projects/:project/jobs one project's jobs +GET /api/v1/jobs/:job +GET /api/v1/projects/:project/jobs/:job +GET /api/v1/jobs/:job/tasks +POST /api/v1/jobs/:job/cancel +POST /api/v1/jobs/:job/retry +``` + +A job id is unique on its own, so naming the project is optional. Where it +is named it is asserted: a job of some other project is a 404 rather than +being quietly served. The lists take `?state=` alongside the paging. + +```json +{ + "ok": true, + "data": { "id": "...", "number": 12, "state": "running", "ref": "refs/heads/main", + "head_sha": "...", "visibility": "private", "created_at": 1800000000000 } +} ``` -Returns the job and the state of every task in it. +Three ways to be allowed: the job is public, the caller holds the project's +trigger secret, or the caller may manage the project. One credential +therefore covers starting a build and watching it. + +`/tasks` returns the tasks and the edges between them, which is what a +stage graph is drawn from: ```json { "ok": true, "data": { - "job": { "id": "...", "number": 12, "state": "running", "ref": "refs/heads/main", - "head_sha": "...", "visibility": "private", "created_at": 1800000000000 }, "tasks": [ { "id": "...", "name": "build:arch=x86_64", "state": "success", "exit_code": 0, "arch": "x86_64", "attempt": 1, "worker_name": "builder-1" } + ], + "deps": [ + { "task_id": "...", "depends_on_id": "...", "kind": "needs" } ] }, "pagination": { "offset": 0, "total": 6 } } ``` -`pagination` describes `tasks`. +`kind` is `needs` or `after`. An `after` edge orders a task without putting +it downstream of a failure. -Three ways to be allowed: the job is public, the caller holds the project's -trigger secret, or the caller is signed in and may manage the project. One -credential therefore covers starting a build and watching it. +`cancel` only applies to a running job and is a 409 otherwise. `retry` +starts a new job at the same commit and returns its id; the original stays +readable rather than being overwritten. Reading a task back ------------------- ``` GET /api/v1/tasks/:task -GET /api/v1/projects/:project/tasks/:task +GET /api/v1/tasks/:task/log +GET /api/v1/tasks/:task/outputs +GET /api/v1/tasks/:task/artifacts ``` -A task's state and the artifacts it stored. Both paths return the same -body. A task id is unique on its own, so naming the project is optional; -doing so only widens what may reach a private task, because there is then a -trigger secret to check against. - -| | public task | trigger secret | session that may manage the project | -| --- | --- | --- | --- | -| `/tasks/:task` | yes | not applicable | yes | -| `/projects/:project/tasks/:task` | yes | yes | yes | - -A signature on the bare path proves nothing, since there is no project in -it to look a secret up from. On the scoped path the project must be the one -the task belongs to, so a guessed id cannot be read through some other -project that happens to be readable. Signing a GET means an HMAC over an +Each also answers under `/api/v1/projects/:project/tasks/:task`, where the +project is asserted: a guessed id cannot be read through whichever project +happens to be readable. The task names its own project either way, so a +trigger secret is checked on both. Signing a GET means an HMAC over an empty body. ```json @@ -199,31 +221,36 @@ empty body. "worker_name": "builder-1", "timeout": 3600, "workdir": "/work", "needs": ["build:arch=x86_64"], "after": [], "matrix": { "pkg": "musl" }, "artifact_paths": ["dist/**"], - "outputs": [ - { "name": "tag", "masked": false, "value": "v1.2.3" }, - { "name": "token", "masked": true, "value": null } - ], - "log_size": 4096, + "log_size": 4096, "log_expired": false, "created_at": 1800000000000, "started_at": null, "finished_at": null, "project_id": "demo", "job_id": "m2y5rj7ykdm9sfbr54hf", "job_number": 12, "ref": "refs/heads/main", "sha": "<40 hex>", "visibility": "public" - }, - "artifacts": [ - { "id": "...", "path": "dist/app", "size": 1048576, "sha256": "<64 hex>", - "created_at": 1800000000000, "expires_at": null, - "url": "https://ci.example.com/api/v1/projects/demo/jobs/<job>/tasks/<task>/artifacts/<id>" } - ] - }, - "pagination": { "offset": 0, "total": 1 } + } + } } ``` -`pagination` describes `artifacts`. `artifact_paths` is what the pipeline -told the task to collect; `artifacts` is what it actually stored. Each carries a ready download `url`, which is a -convenience rather than the permission: that route checks visibility again -for itself. +The log, the outputs and the artifacts are each their own page, because +each grows without bound and none of them belongs inside one response. + +``` +GET /api/v1/tasks/:task/log?offset=0&limit=262144 +``` + +```json +{ "ok": true, "data": { "offset": 0, "size": 4096, "data": "...", "expired": false } } +``` + +`size` is the whole log, so a tail asks from where it stopped and is given +only what is new. `expired` is true once retention has removed it, which is +a different answer from an empty log. -`outputs` is what the task wrote with `::set-output`. A masked one reports +`artifact_paths` on the task is what the pipeline told it to collect; +`/artifacts` is what it actually stored. Each carries a ready download +`url`, a convenience rather than the permission: that route checks +visibility again for itself. + +`/outputs` is what the task wrote with `::set-output`. A masked one reports its name so the shape of a run is visible, and never its value. There is no `env` and no `services` here, deliberately. Both can hold @@ -232,6 +259,64 @@ project variables, and a service carries an environment of its own. The repository that is not public. A worker gets all three because it has to; nobody else does. +Managing things +--------------- + +All of this needs an access token. What it may reach follows from who the +token belongs to rather than from which URL was used: an administrator acts +on anything, anyone else on what they own. Something they may not see is +reported as missing, not refused. + +``` +GET /api/v1/users /users/:id PATCH, DELETE POST /users +GET /api/v1/projects /projects/:id PATCH, DELETE POST /projects +GET /api/v1/workers /workers/:id PATCH, DELETE POST /workers +GET /api/v1/sessions /sessions/:id DELETE +``` + +`me` is an ordinary user id that resolves to the caller, so `GET +/api/v1/users/me` is who am I. Listing users as a non-administrator returns +just themselves rather than a refusal. + +Two things are shown exactly once, in the response that creates them, and +are not readable afterwards: + +```json +{ "ok": true, "data": { "id": "demo", "trigger_secret": "<64 hex>" } } +{ "ok": true, "data": { "id": "...", "name": "builder-1", "token": "<64 hex>" } } +``` + +`POST /api/v1/projects/:id/trigger-secret` replaces the first. There is no +way to read either back; losing one means generating another. + +Project retention takes three answers and they are all different. Absent +leaves the setting alone, `null` follows the server default, and `0` keeps +forever: + +``` +PATCH /api/v1/projects/:id { "artifact_keep_days": null, "log_keep_days": 0 } +``` + +Variables are per project, and a value is never read back: + +``` +GET /api/v1/projects/:id/variables +PUT /api/v1/projects/:id/variables/:name { "value": "...", "masked": true } +DELETE /api/v1/projects/:id/variables/:name +``` + +A worker's machines and its recent tasks hang off it: + +``` +GET /api/v1/workers/:id/agents +GET /api/v1/workers/:id/tasks +``` + +Sessions are the revocation surface. Each row carries the browser and +address it was last used from, so a person can tell one from another. They +are recorded for that and nothing else: a session arriving from a new +address is ordinary, and is never refused for it. + The worker API -------------- diff --git a/docs/auth.md b/docs/auth.md @@ -0,0 +1,196 @@ +Authentication +============== + +The conductor has one way in and two providers behind it. Either it runs +its own login page, or an external provider does. Never both: the discovery +document can only name one, and whichever it names is the one that answers. + +The application never learns which it is talking to. It asks for the +discovery document, runs an authorization code flow against whatever that +names, and presents the resulting access token as a bearer credential. + +Discovery +--------- + +``` +GET /.well-known/openid-configuration +``` + +Served in both modes, on the conductor's own origin. With a provider +configured this is that provider's document, passed through. Without one it +describes the conductor's own endpoints. + +One non-standard field is added either way: + +```json +{ "issuer": "...", "authorization_endpoint": "...", "token_endpoint": "...", + "client_id": "conductor", "scopes": "openid" } +``` + +A discovery document carries no client id, and the application has to learn +one somewhere. Putting it here saves a second request and a second thing to +configure. + +The flow +-------- + +An ordinary authorization code flow with PKCE, which is the only kind the +conductor accepts. `code_challenge_method` must be `S256`; there is no +`plain` and no implicit grant. + +``` +1. GET /.well-known/openid-configuration +2. redirect to authorization_endpoint with code_challenge +3. the provider sends the browser back with ?code= +4. POST token_endpoint with code_verifier +5. Authorization: Bearer <access_token> from then on +``` + +The redirect uri is the application's own origin and path, with no +fragment. The fragment belongs to the router, so the code comes back on the +query string instead. + +Tokens +------ + +An access token lives in memory only and is short-lived, `auth.access_ttl`, +900 seconds by default. It is a JWT either way: signed by the provider, or +signed by the conductor with `auth.session_secret`. + +A refresh token is not a JWT. It is an opaque random value, stored only as +a sha256 hash, and it is the session. It lives in local storage so a +reload does not sign you out. + +The refresh token is not rotated. Rotation only catches a stolen token in +the case where the victim then presents the superseded copy, and in +exchange two tabs refreshing at once race into a spurious logout and a +hand-revoked session cannot be told from a leaked one. Revocation is the +control that matters here, and it is explicit. + +`auth.refresh_sliding` decides what a refresh does to the deadline. True, +the default, moves it to `now + auth.refresh_ttl`, so a session lasts that +long after it was last used. False leaves the original deadline alone, so a +session ends that long after signing in whatever happens in between. + +Revoking +-------- + +``` +GET /api/v1/sessions +DELETE /api/v1/sessions/:id +POST /auth/revoke +``` + +Every live session is listed with the browser and address it was last seen +from, so a person can tell them apart. Those two are recorded for that and +nothing else. They are never compared and never validated: moving between +networks is ordinary, and a browser that updates itself is not an intruder. + +An administrator sees every session, anyone else sees their own. Revoking +one ends it at the next refresh, which is at most `auth.access_ttl` away. +Disabling an account or changing its password revokes all of its sessions +at once. + +With a provider configured none of this is populated. Nothing issues a +refresh token, and a session is ended in the provider. + +The built-in provider +--------------------- + +Used when no `auth.oidc.discovery_url` is set, or when +`auth.local.enabled` is set true to override one. + +``` +GET /.well-known/openid-configuration +GET /auth/authorize the login page +POST /auth/authorize credentials, redirects with a code +POST /auth/token authorization_code and refresh_token grants +POST /auth/revoke +GET /auth/logout +GET /auth/jwks an empty set +``` + +The login page is a static file with a plain form. It has no action, so the +authorization parameters stay in the query string where the page was asked +for, and nothing has to be rendered into it. There is no inline script and +no inline style, which is what lets a Content-Security-Policy ship with the +application at all. + +Its key set is empty and honestly so. The tokens are HS256 and nothing +outside this conductor verifies them; the endpoint exists for shape. + +An authorization code is a signed token that lives for 60 seconds. It +carries the subject, the redirect uri, the code challenge and an id, so +none of that needs storing. Redeeming one writes the session row under an +id derived from the code's own id, so redeeming it twice collides on the +primary key and is refused. Codes are single use without a table of spent +codes. + +`auth.local.redirect_uris` is where the browser may be sent back to. It +defaults to `server.public_url` and is compared by origin, so an +application served from somewhere else needs its origin listed. A redirect +uri that is not listed is refused outright rather than redirected to, or +this would be an open redirect. + +Using an external provider +-------------------------- + +Set the discovery url and the client id: + +```yaml +auth: + oidc: + discovery_url: https://idp.example.com/realms/ci/.well-known/openid-configuration + client_id: conductor + admin_role: conductor-admin +``` + +There is no client secret. The application runs the flow itself as a public +client, so the conductor never redeems a code and has nothing to +authenticate with. A confidential client cannot be used here. + +In Keycloak that means: + +- the client is **public**, not confidential +- **Standard flow** on, with **PKCE** required and `S256` as the method +- the application's origin in **Web Origins**, or the browser cannot reach + the token endpoint +- the application's origin and path as a **Valid redirect URI** + +Roles are read from `realm_access.roles`, `resource_access[*].roles`, a +flat `roles` claim and `groups`, because there is no standard. Anyone +holding `auth.oidc.admin_role` is an administrator here. + +A provider account still gets a local row on first sight, because projects +and worker tokens reference `users(id)`. Identity is the issuer and subject +together, not the display name, so renaming someone at the provider does +not produce a second account. That row can be disabled locally, which locks +them out even though the provider would still admit them. + +The audience trap +----------------- + +For a client named `conductor`, Keycloak issues an id_token with +`aud: "conductor"` but an access token with `aud: ["account"]` and +`azp: "conductor"`. The conductor checks `azp`, falling back to `aud` +containing the client id. Checking only `aud` rejects every real access +token, and the failure is a 401 on every request. + +This is worth knowing because it does not show up in testing unless the +mock provider is made to reproduce it. `test/helpers/oidc.js` has a persona +that does. + +Workers and triggers +-------------------- + +Neither goes through any of this. + +A worker token is 64 hex characters, presented as a bearer credential and +stored as a sha256 hash. It never parses as a JWT, so the two meanings of +the header cannot be confused. It reaches the worker protocol and nothing +else. + +A project trigger secret is an HMAC over the exact request body, in +`X-Hub-Signature-256`, or `X-Gitlab-Token` compared directly. A project +with no secret accepts unsigned triggers and says so in the log, which is +convenient on a private network and a bad idea anywhere else. diff --git a/docs/deployment.md b/docs/deployment.md @@ -2,8 +2,9 @@ Deployment ========== One service, one image. The conductor accepts triggers, schedules jobs, -serves the worker API and renders the interface. Workers are separate and -may live anywhere, including on machines you do not administer. +serves the API and hosts the application that drives it. Workers are +separate and may live anywhere, including on machines you do not +administer. Quick start ----------- @@ -92,7 +93,7 @@ Worth setting in production: | Variable | Why | | -------------------------- | ------------------------------------------------ | | `CONDUCTOR_PUBLIC_URL` | Workers get absolute callback URLs built from it. | -| `CONDUCTOR_SESSION_SECRET` | Otherwise sessions end at every restart. | +| `CONDUCTOR_SESSION_SECRET` | Otherwise access tokens stop verifying at every restart. | | `CONDUCTOR_SECRET_KEY` | Otherwise stored secrets are kept in the clear. | | `CONDUCTOR_ADMIN_PASSWORD` | Otherwise one is generated and logged once. | @@ -172,7 +173,32 @@ artifact to its own disk before the conductor sees a byte, which for multi gigabyte build output is both slow and surprising. Set `CONDUCTOR_PUBLIC_URL=https://ci.example.com` to match, or workers will -be handed callback URLs that do not work. +be handed callback URLs that do not work. It is also the issuer the +built-in provider names in its tokens, and the default redirect uri the +application is allowed to come back to. + +Nothing special is needed for the application's own routing. It routes on +the fragment, so a deep link is `https://ci.example.com/#!/jobs/<id>` and +the path the proxy sees is always the root. There is no history fallback +to configure. + +### Serving the application from somewhere else + +The bundle in `web/dist` is static and can be served by anything. If it is +not on the conductor's own origin, two things have to be said: + +```sh +CONDUCTOR_LOCAL_REDIRECT_URIS=https://ci.example.com,https://app.example.com +``` + +so the built-in provider will send a browser back there, and the +application still needs to reach `/.well-known/openid-configuration` and +`/api/v1` on the conductor. CORS is already open, but the browser will +refuse the cross-origin fetch unless the Content-Security-Policy on +whatever serves the bundle allows it. + +Hosting it on the conductor itself avoids all of that, which is why that is +the default. Retention --------- @@ -221,8 +247,8 @@ job. That is deliberate, and is how a bulky intermediate avoids becoming immortal by being green. Logs go by age alone. Sweeping a log deletes the bytes and leaves the task, -so a job stays explainable after its output is gone, and the interface can -tell an expired log apart from one that was never written. +so a job stays explainable after its output is gone, and a reader is told +the log expired rather than that it was empty. The same sweep forgets machines that have stopped claiming, after `retention.worker_agent_keep_hours` (default 24, `0` keeps them for ever). diff --git a/test/self-pipeline.test.js b/test/self-pipeline.test.js @@ -29,7 +29,7 @@ test('its tasks declare the features they actually need', async () => { const { tasks } = await selfPipeline(); const byName = new Map(tasks.map((task) => [task.name, task])); - for (const name of ['style', 'test', 'integration', 'images', 'smoke', 'publish']) { + for (const name of ['style', 'build', 'test', 'integration', 'images', 'smoke', 'publish']) { assert.ok(byName.has(name), `expected a ${name} task`); } @@ -47,7 +47,7 @@ test('its tasks declare the features they actually need', async () => { } // Conversely, the tasks meant to run anywhere must not demand features. - for (const name of ['style', 'test']) { + for (const name of ['style', 'build', 'test']) { assert.deepEqual(byName.get(name).requires, [], `${name} should run on any worker`); } });