conductor

CI task system
git clone git://git.finwo.net/app/conductor
Log | Files | Refs | README | LICENSE

commit f7f27db73b4a7d99e7db20e970c24d3ba9629fab
parent 8459c720158b79b0f18bf25e9b3374a59338eac9
Author: finwo <finwo@pm.me>
Date:   Sun, 20 Sep 2026 03:24:39 +0200

Document jobs, tasks and the endpoints that replaced the worker api

Diffstat:
M.conductor.yml | 2+-
MREADME.md | 34+++++++++++++++++-----------------
Mconductor.example.yaml | 16++++++++--------
Mdeploy/hub/conductor-worker.md | 37+++++++++++++++++++------------------
Mdeploy/hub/conductor.md | 17+++++++++--------
Mdeploy/smoke.sh | 58++++++++++++++++++++++++++++++----------------------------
Mdocs/api.md | 186++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++-------------------
Mdocs/deployment.md | 40++++++++++++++++++++--------------------
Mdocs/pipeline.md | 103+++++++++++++++++++++++++++++++++++++++++++------------------------------------
Mdocs/worker.md | 67++++++++++++++++++++++++++++++++++---------------------------------
Mexamples/distro.conductor.yml | 2+-
Mexamples/node-app.conductor.yml | 2+-
12 files changed, 338 insertions(+), 226 deletions(-)

diff --git a/.conductor.yml b/.conductor.yml @@ -13,7 +13,7 @@ defaults: image: node:24-bookworm-slim timeout: 15m -jobs: +tasks: # Style rules that are cheap to break and cheap to check: ASCII only, # no CRLF, no trailing whitespace. style: diff --git a/README.md b/README.md @@ -1,19 +1,19 @@ conductor ========= -A stand-alone CI system. Pipelines live in the repository, jobs run in +A stand-alone CI system. Pipelines live in the repository, tasks run in containers, and workers pull work rather than being pushed to. Summary ------- The conductor accepts a push notification, reads `.conductor.yml` at the -pushed commit, and turns it into a graph of jobs. Workers poll for work, -advertising the architectures and features they can offer. Any job whose +pushed commit, and turns it into a graph of tasks. Workers poll for work, +advertising the architectures and features they can offer. Any task whose dependencies have succeeded is eligible, so a wide graph runs wide. Workers need no repository credentials: the conductor serves the source tree -for the exact commit a job was handed. A worker on someone else's hardware +for the exact commit a task was handed. A worker on someone else's hardware can therefore build for you without being able to read anything else. Nothing has to be set up before it will run. With no configuration it uses @@ -29,8 +29,8 @@ Under construction. Working today: - configuration, with sqlite, mysql/tidb and postgres backends - local and S3 compatible object storage - pipeline parsing, matrix and architecture expansion, dependency graphs - - triggers, run scheduling, the worker API, live logs, artifacts, reaping - - the worker: containerised jobs, services, features, caches, cleanup + - triggers, job scheduling, the worker API, live logs, artifacts, reaping + - the worker: containerised tasks, services, features, caches, cleanup - accounts, OIDC, project ownership, visibility and project variables - the web interface, server rendered with htmx - container images for amd64, arm64 and riscv64, published on release @@ -120,7 +120,7 @@ See [docs/worker.md](docs/worker.md), and `examples/worker.json`. Who sees what ------------- -Everything is private by default. A run is visible to an anonymous visitor +Everything is private by default. A job is visible to an anonymous visitor only when the project, or the pipeline at that commit, says it is public: ```yaml @@ -128,17 +128,17 @@ version: 1 visibility: public ``` -Because it is recorded per run, making a repository private stops exposing -future runs without rewriting the history of past ones. +Because it is recorded per job, making a repository private stops exposing +future jobs without rewriting the history of past ones. Users register their own projects and their own workers. A worker someone -registers is only ever offered jobs from that person's projects, which is +registers is only ever offered tasks from that person's projects, which is what makes it safe to accept build capacity from people you do not otherwise trust. A worker created by an administrator with no owner is shared, and runs anything. Administrators see and manage everything, and manage accounts. Deleting an -account deletes the projects and workers it owned, along with their run +account deletes the projects and workers it owned, along with their job history: leaving them behind would turn a personal worker into a shared one. Pipelines @@ -151,7 +151,7 @@ defaults: image: debian:bookworm-slim timeout: 30m -jobs: +tasks: test: script: - make test @@ -178,13 +178,13 @@ jobs: - ./publish.sh ``` -`arch` and `matrix` fan a job out into one job per combination. A dependency +`arch` and `matrix` fan a task out into one task per combination. A dependency that shares a dimension is matched on it, so the aarch64 package waits for the aarch64 build rather than for every build, while `publish`, which has no architecture of its own, waits for all of them. `requires` names features a worker must offer. A worker holding a signing key -advertises `sign-key`, and only jobs asking for it are sent there. The key +advertises `sign-key`, and only tasks asking for it are sent there. The key stays on the worker and is never known to the conductor. See [docs/pipeline.md](docs/pipeline.md) for the full reference, @@ -196,7 +196,7 @@ Architecture ------------ ``` -git push -> post-receive -> conductor -> job graph in the database +git push -> post-receive -> conductor -> task graph in the database | worker polls | @@ -204,7 +204,7 @@ git push -> post-receive -> conductor -> job graph in the database ``` - `src/conductor` triggers, scheduling, the worker API, the interface - - `src/worker` the agent that runs jobs, with no npm dependencies + - `src/worker` the agent that runs tasks, with no npm dependencies - `src/lib` configuration, database, storage, pipelines, git Queries are written once against all three databases using named `{key}` @@ -223,7 +223,7 @@ needs and cleans up after itself: | Dependency | Used for | | ------------------- | -------------------------------------------- | -| `alpine` | containers for the worker to actually run jobs in | +| `alpine` | containers for the worker to actually run tasks in | | MinIO | object storage, artifacts and archived logs | | mock-oauth2-server | a real OIDC provider to authenticate against | diff --git a/conductor.example.yaml b/conductor.example.yaml @@ -82,16 +82,16 @@ log: # its settings page; zero means keep forever. # # Artifacts follow two rules and survive if either wants them: the last -# artifact_keep_runs runs are kept whatever their age, and anything younger -# than artifact_keep_days is kept however many runs have followed. The most -# recent successful run is always kept, so a project that has gone quiet -# still has something to download. A job that sets artifacts.expire in the +# artifact_keep_jobs jobs are kept whatever their age, and anything younger +# than artifact_keep_days is kept however many jobs have followed. The most +# recent successful job is always kept, so a project that has gone quiet +# still has something to download. A task that sets artifacts.expire in the # pipeline overrides all of it with an exact deadline. # # Logs go by age alone. They are the reason this exists: written for every -# job, read for almost none, and otherwise never removed. +# task, read for almost none, and otherwise never removed. retention: - artifact_keep_runs: 10 + artifact_keep_jobs: 10 artifact_keep_days: 30 log_keep_days: 14 # Seconds between sweeps, and how much one sweep will delete, so a first @@ -100,9 +100,9 @@ retention: batch: 500 scheduler: - # A claimed job whose worker stops reporting for this long is treated as + # A claimed task whose worker stops reporting for this long is treated as # lost, then retried or failed. Must exceed reap_interval. heartbeat_timeout: 120 reap_interval: 30 - default_job_timeout: 3600 + default_task_timeout: 3600 max_attempts: 3 diff --git a/deploy/hub/conductor-worker.md b/deploy/hub/conductor-worker.md @@ -1,7 +1,7 @@ # conductor-worker -Runs jobs for a [conductor](https://hub.docker.com/r/finwo/conductor) -server. Polls for work, runs each job in its own container, streams the +Runs tasks for a [conductor](https://hub.docker.com/r/finwo/conductor) +server. Polls for work, runs each task in its own container, streams the log back and uploads artifacts. Source and issues: https://github.com/finwo/conductor @@ -11,16 +11,17 @@ Source and issues: https://github.com/finwo/conductor | Tag | What it is | | ---------------- | ------------------------------------------ | | `latest` | the most recent release | -| `0.2.0` | an exact version | -| `0.2` | the latest patch of that minor series | +| `0.3.0` | an exact version | +| `0.3` | the latest patch of that minor series | | `<commit>` | an exact build from main, twelve hex chars | Built for `linux/amd64`, `linux/arm64` and `linux/riscv64`. -**Upgrade the conductor and its workers together.** 0.2.0 changed how a -job's tree reaches its container, and a 0.1.x worker will not run jobs -from a 0.2.x conductor. Pin both to the same version if they are -upgraded separately. +**Upgrade the conductor and its workers together.** In 0.3.0 a job became +one run of a pipeline and a task became the unit this image runs. Workers +claim from `/api/v1/tasks/claim`, so a 0.2.x worker will not run anything +for a 0.3.x conductor. Pin both to the same version if they are upgraded +separately. ## Quick start @@ -38,21 +39,21 @@ docker run -d \ finwo/conductor-worker ``` -A worker you register is only ever offered jobs from your own projects. +A worker you register is only ever offered tasks from your own projects. ## It keeps nothing -The socket is the only thing this image needs. A job's tree is unpacked -into the job's own container over that socket, and its artifacts are read +The socket is the only thing this image needs. A task's tree is unpacked +into the task's own container over that socket, and its artifacts are read back the same way, so the worker shares no directory with what it runs. That means no volumes, nothing to persist between restarts, and no paths that have to mean the same thing inside and outside the container. The -job containers are siblings on the host daemon rather than children, but +task containers are siblings on the host daemon rather than children, but since nothing of the worker's filesystem is handed to them, that stops being something you have to think about. -Where a job runs is decided by the build, not by this machine: the +Where a task runs is decided by the build, not by this machine: the `workdir` key in the pipeline, the project's setting, or `/work`. ## Configuration @@ -61,8 +62,8 @@ Where a job runs is decided by the build, not by this machine: the | ------------------------------ | -------------- | ----------------------------------- | | `CONDUCTOR_URL` | required | Where the conductor is. | | `CONDUCTOR_WORKER_TOKEN` | required | Issued by the conductor, shown once. | -| `CONDUCTOR_WORKER_NAME` | the hostname | How this worker appears in runs. | -| `CONDUCTOR_WORKER_CONCURRENCY` | `1` | Jobs at once. | +| `CONDUCTOR_WORKER_NAME` | the hostname | How this worker appears in tasks. | +| `CONDUCTOR_WORKER_CONCURRENCY` | `1` | Tasks at once. | | `CONDUCTOR_WORKER_ARCHES` | any | Architectures offered, comma separated. | | `CONDUCTOR_WORKER_TOKEN_FILE` | none | Read the token from a file instead. | | `CONDUCTOR_WORKER_DOCKER` | `docker` | Runtime CLI: docker, podman, nerdctl. | @@ -74,8 +75,8 @@ is what you want unless one worker serves several. ## Features A feature is a name this worker advertises, together with the local -resources a job asking for it should be given. A pipeline job asks with -`requires:`, and the conductor will not send that job to a worker which +resources a task asking for it should be given. A pipeline task asks with +`requires:`, and the conductor will not send that task to a worker which does not advertise every name it lists. The conductor only ever learns the names. @@ -103,7 +104,7 @@ configuration file, JSON or YAML, mounted into the container: A file at that path is picked up on its own, with the token still coming from the environment if you would rather keep it out of the file. Nothing enforces what a name means beyond matching it, so `docker` is -simply how you say this worker's jobs may use the socket mounted above. +simply how you say this worker's tasks may use the socket mounted above. ## Documentation diff --git a/deploy/hub/conductor.md b/deploy/hub/conductor.md @@ -1,6 +1,6 @@ # conductor -A small CI server. Accepts triggers, schedules runs, serves the worker +A small CI server. Accepts triggers, schedules jobs, serves the worker API and renders the interface. One service, one image. Source and issues: https://github.com/finwo/conductor @@ -10,15 +10,16 @@ Source and issues: https://github.com/finwo/conductor | Tag | What it is | | ---------------- | ------------------------------------------ | | `latest` | the most recent release | -| `0.2.0` | an exact version | -| `0.2` | the latest patch of that minor series | +| `0.3.0` | an exact version | +| `0.3` | the latest patch of that minor series | | `<commit>` | an exact build from main, twelve hex chars | Built for `linux/amd64`, `linux/arm64` and `linux/riscv64`. -**Upgrade the conductor and its workers together.** 0.2.0 changed how a -job's tree reaches its container, and a 0.1.x worker will not run jobs -from a 0.2.x conductor. +**Upgrade the conductor and its workers together.** In 0.3.0 a job became +one run of a pipeline and a task became the unit a worker runs. Workers +claim from `/api/v1/tasks/claim`, so a 0.2.x worker will not run anything +for a 0.3.x conductor. ## Quick start @@ -34,7 +35,7 @@ docker run -d \ The administrator password is printed once in the log on first start. Set `CONDUCTOR_ADMIN_PASSWORD` to choose it instead. -Jobs need a worker, which is a separate image: see +Tasks need a worker, which is a separate image: see [finwo/conductor-worker](https://hub.docker.com/r/finwo/conductor-worker). A compose file running both is in the repository under `deploy/`. @@ -85,7 +86,7 @@ handed over with a presigned redirect rather than proxied. Build output is deleted on a schedule, or a busy server fills its disk with logs nobody will read. The defaults keep artifacts for 30 days or -the last 10 runs, and logs for 14 days, and a project may override any of +the last 10 jobs, and logs for 14 days, and a project may override any of it. Upgrading an installation that has been running for a while will delete a lot on the first sweep, so set these before starting if that matters: diff --git a/deploy/smoke.sh b/deploy/smoke.sh @@ -52,10 +52,10 @@ version: 1 visibility: public defaults: image: alpine:3 -jobs: +tasks: build: script: - - echo "run $CONDUCTOR_RUN_NUMBER of $CONDUCTOR_PROJECT" + - echo "run $CONDUCTOR_JOB_NUMBER of $CONDUCTOR_PROJECT" - mkdir -p out && echo "packaged" > out/result.txt artifacts: paths: [out/**] @@ -127,7 +127,7 @@ export SMOKE_TOKEN log "starting the worker" compose up -d worker >/dev/null 2>&1 -log "triggering a run" +log "triggering a job" BODY=$(printf '{"sha":"%s","ref":"refs/heads/main"}' "${SHA}") SIG=$(printf '%s' "${BODY}" | openssl dgst -sha256 -hmac "${SECRET}" | sed 's/^.*[= ]//') RESPONSE=$(curl -fsS -X POST "http://127.0.0.1:${PORT}/api/v1/projects/demo/trigger" \ @@ -136,21 +136,20 @@ RESPONSE=$(curl -fsS -X POST "http://127.0.0.1:${PORT}/api/v1/projects/demo/trig -d "${BODY}") printf '%s\n' "${RESPONSE}" -RUN=$(printf '%s' "${RESPONSE}" | sed 's/.*"run_id":"\([^"]*\)".*/\1/') -[ -n "${RUN}" ] || fail "no run was created" +JOB=$(printf '%s' "${RESPONSE}" | sed 's/.*"job_id":"\([^"]*\)".*/\1/') +[ -n "${JOB}" ] || fail "no job was created" -# The run state is the first badge on the run page. There is no read API; -# the interface is the only place to see it. -run_state() { - curl -fsS "http://127.0.0.1:${PORT}/runs/$1" \ +# The job state is the first badge on the job page. +job_state() { + curl -fsS "http://127.0.0.1:${PORT}/jobs/$1" \ | sed -n 's/.*class="badge \([a-z]*\)".*/\1/p' | head -1 } -log "waiting for the run to finish" +log "waiting for the job to finish" STATE_NOW= i=0 while [ "${i}" -lt 90 ]; do - STATE_NOW=$(run_state "${RUN}") + STATE_NOW=$(job_state "${JOB}") case "${STATE_NOW}" in success|failed|cancelled) break ;; esac @@ -158,18 +157,21 @@ while [ "${i}" -lt 90 ]; do sleep 1 done -printf 'run %s finished as %s after %ss\n' "${RUN}" "${STATE_NOW}" "${i}" +printf 'job %s finished as %s after %ss\n' "${JOB}" "${STATE_NOW}" "${i}" [ "${STATE_NOW}" = success ] || { compose logs worker | tail -30 - fail "the run ended as ${STATE_NOW}" + fail "the job ended as ${STATE_NOW}" } log "checking the log and the artifact" -JOB="${RUN}:build" -ENCODED=$(printf '%s' "${JOB}" | sed 's/:/%3A/g') +# Task ids carry no structure, so the build task is found by following the +# link from the job page rather than by assembling an id. +TASK=$(curl -fsS "http://127.0.0.1:${PORT}/jobs/${JOB}" \ + | sed -n 's#.*href="/tasks/\([^"]*\)">build<.*#\1#p' | head -1) +[ -n "${TASK}" ] || fail "no build task was linked from the job page" -PAGE=$(curl -fsS "http://127.0.0.1:${PORT}/jobs/${ENCODED}") -printf '%s\n' "${PAGE}" | grep -q 'run 1 of demo' || fail "the job environment did not reach the script" +PAGE=$(curl -fsS "http://127.0.0.1:${PORT}/tasks/${TASK}") +printf '%s\n' "${PAGE}" | grep -q 'run 1 of demo' || fail "the task environment did not reach the script" ARTIFACT_PATH=$(printf '%s' "${PAGE}" \ | sed -n 's#.*href="\(/api/v1/[^"]*/artifacts/[^"]*\)".*#\1#p' | head -1) @@ -182,10 +184,10 @@ printf 'artifact contents: %s\n' "${CONTENT}" log "checking the interface" curl -fsS "http://127.0.0.1:${PORT}/" | grep -q 'conductor' || fail "the interface did not render" -# The release job is restricted to main. A push to anything else must not +# The release task is restricted to main. A push to anything else must not # produce it at all, rather than produce it and skip it, since a skipped -# job fails the run. -log "checking that a branch push leaves the restricted job out" +# task fails the job. +log "checking that a branch push leaves the restricted task out" BRANCH_BODY=$(printf '{"sha":"%s","ref":"refs/heads/feature"}' "${SHA}") BRANCH_SIG=$(printf '%s' "${BRANCH_BODY}" | openssl dgst -sha256 -hmac "${SECRET}" | sed 's/^.*[= ]//') BRANCH=$(curl -fsS -X POST "http://127.0.0.1:${PORT}/api/v1/projects/demo/trigger" \ @@ -194,13 +196,13 @@ BRANCH=$(curl -fsS -X POST "http://127.0.0.1:${PORT}/api/v1/projects/demo/trigge -d "${BRANCH_BODY}") printf '%s\n' "${BRANCH}" -printf '%s' "${BRANCH}" | grep -q '"jobs":2' \ - || fail "expected 2 jobs on a branch, got: ${BRANCH}" +printf '%s' "${BRANCH}" | grep -q '"tasks":2' \ + || fail "expected 2 tasks on a branch, got: ${BRANCH}" -BRANCH_RUN=$(printf '%s' "${BRANCH}" | sed 's/.*"run_id":"\([^"]*\)".*/\1/') +BRANCH_JOB=$(printf '%s' "${BRANCH}" | sed 's/.*"job_id":"\([^"]*\)".*/\1/') i=0 while [ "${i}" -lt 90 ]; do - BRANCH_STATE=$(run_state "${BRANCH_RUN}") + BRANCH_STATE=$(job_state "${BRANCH_JOB}") case "${BRANCH_STATE}" in success|failed|cancelled) break ;; esac @@ -208,10 +210,10 @@ while [ "${i}" -lt 90 ]; do sleep 1 done -curl -fsS "http://127.0.0.1:${PORT}/runs/${BRANCH_RUN}" | grep -q '>release<' \ - && fail "the release job should not exist on a branch run" +curl -fsS "http://127.0.0.1:${PORT}/jobs/${BRANCH_JOB}" | grep -q '>release<' \ + && fail "the release task should not exist on a branch job" -printf 'branch run finished as %s without the release job\n' "${BRANCH_STATE}" -[ "${BRANCH_STATE}" = success ] || fail "the branch run ended as ${BRANCH_STATE}" +printf 'branch job finished as %s without the release task\n' "${BRANCH_STATE}" +[ "${BRANCH_STATE}" = success ] || fail "the branch job ended as ${BRANCH_STATE}" printf '\nPASSED\n' diff --git a/docs/api.md b/docs/api.md @@ -1,22 +1,41 @@ HTTP API ======== -The conductor exposes exactly three HTTP surfaces, all under `/api/v1`. -Everything else, including signing in and managing projects, workers and -users, happens in the interface. +Two words to get straight first, because everything below turns on them. -| Surface | Authentication | -| ------------------------------------ | --------------------- | -| `/api/v1/projects/:project/trigger` | per project HMAC | -| `/api/v1/workers/jobs` | worker token | -| `/api/v1/projects/.../artifacts/...` | session, or anonymous for a public run | +A **job** is one run of a pipeline. Creating it compiles the repository's +`.conductor.yml` at a commit, which produces one or more **tasks**. A task +is the standalone unit a worker runs: one container, one script, one +result. + +A worker deals only in tasks. It is never told which project or job a task +belongs to, and the task id it receives carries no structure it could read +that out of. What it gets is a context and a script. + +The conductor exposes four HTTP surfaces, all under `/api/v1`. Everything +else, including signing in and managing projects, workers and users, +happens in the interface. + +| Surface | Authentication | +| ------------------------------------------ | --------------------- | +| `POST /api/v1/projects/:project/trigger` | per project HMAC | +| `POST /api/v1/projects/:project/jobs` | per project HMAC | +| `GET /api/v1/projects/:project/jobs/:job` | HMAC, session, or anonymous for a public job | +| `/api/v1/tasks/...` | worker token | +| `GET /api/v1/projects/.../artifacts/...` | session, or anonymous for a public job | Errors are `{"error": "..."}` with a meaningful status. A resource the caller may not see returns 404 rather than 403, so absence and denial are indistinguishable. -Triggers --------- +Starting a job +-------------- + +Two ways in, differing only in the shape of what they accept. Both take the +same signature, and both read the pipeline from the repository at the +commit rather than trusting anything the caller says about it. + +### From a push ``` POST /api/v1/projects/:project/trigger @@ -31,41 +50,120 @@ GitHub, Gitea and GitLab. Signed with the project's trigger secret: { "sha": "<40 hex>", "base": "<40 hex or empty>", "ref": "refs/heads/main", "actor": "alice" } ``` -Nothing in the payload is trusted beyond which commit to read. The pipeline -comes from the repository at that commit. +A push that deletes a branch is reported as ignored rather than refused, +since there is nothing there to build. -| Status | Meaning | -| ------ | --------------------------------------------- | -| 200 | run created, or ignored for a branch deletion | -| 401 | signature missing or wrong | -| 404 | no such project | -| 409 | project disabled | -| 422 | the pipeline at that commit is invalid | +### Deliberately -The worker API --------------- +``` +POST /api/v1/projects/:project/jobs +``` + +For anything driving the conductor on purpose rather than forwarding a +hook. No payload archaeology: `sha` is required and must be a full commit +id, and a missing one is an error rather than something to infer. + +```json +{ "sha": "<40 hex>", "ref": "refs/heads/main", "base": null, "actor": "alice", "trigger": "api" } +``` + +Only `sha` is required. `trigger` is recorded as-is and defaults to `api`, +so a job started this way stays distinguishable from one a hook forwarded. + +Both return: + +```json +{ "status": "created", "job_id": "m2y5rj7ykdm9sfbr54hf", "tasks": 6 } +``` + +| Status | Meaning | +| ------ | ---------------------------------------------- | +| 200 | job created, or ignored for a branch deletion | +| 400 | no usable commit in the payload | +| 401 | signature missing or wrong | +| 404 | no such project | +| 409 | project disabled | +| 422 | the pipeline at that commit is invalid | + +A 422 carries `problems`, an array of `{path, message}`, so a broken +pipeline says which key is wrong rather than only that something is. + +Reading a job back +------------------ + +``` +GET /api/v1/projects/:project/jobs/:job +``` + +Returns the job and the state of every task in it. + +```json +{ + "job": { "id": "...", "number": 12, "state": "running", "ref": "refs/heads/main", + "head_sha": "...", "visibility": "private", "created_at": 1800000000000 }, + "tasks": [ + { "id": "...", "name": "build:arch=x86_64", "state": "success", "exit_code": 0, + "arch": "x86_64", "attempt": 1, "worker_name": "builder-1" } + ] +} +``` + +Three ways to be allowed: the job is public, the caller holds the project's +trigger secret, or the caller is signed in and may manage the project. One +credential therefore covers starting a build and watching it. + +The task API +------------ Documented because a worker is a normal client of it, and anyone may write another. Every call needs `Authorization: Bearer <worker token>`, and a -worker may only touch a job it currently holds. - -| Method | Path | Purpose | -| ------ | ----------------------------------------------- | --------------------------- | -| POST | `/api/v1/workers/jobs` | claim work, 204 when idle | -| GET | `/api/v1/workers/jobs/:id/source.tar.gz` | the tree at the commit | -| POST | `/api/v1/workers/jobs/:id/log` | append output | -| POST | `/api/v1/workers/jobs/:id/artifacts` | upload one file | -| POST | `/api/v1/workers/jobs/:id/heartbeat` | stay alive, learn of cancellation | -| POST | `/api/v1/workers/jobs/:id/complete` | report the outcome | - -Claim with a JSON body: `{"arches":["x86_64","aarch64"],"features":["dind","sign-key"],"name":"my-worker"}`. -Only jobs whose architecture the worker offers and whose `requires` it +worker may only touch a task it currently holds. + +| Method | Path | Purpose | +| ------ | -------------------------------------- | --------------------------------- | +| POST | `/api/v1/tasks/claim` | claim work, 204 when idle | +| GET | `/api/v1/tasks/:task/source.tar.gz` | the tree at the commit | +| POST | `/api/v1/tasks/:task/log` | append output | +| POST | `/api/v1/tasks/:task/artifacts` | upload one file | +| POST | `/api/v1/tasks/:task/heartbeat` | stay alive, learn of cancellation | +| POST | `/api/v1/tasks/:task/complete` | report the outcome | + +Claim with a JSON body: +`{"arches":["x86_64","aarch64"],"features":["dind","sign-key"],"name":"my-worker"}`. +Only tasks whose architecture the worker offers and whose `requires` it satisfies are handed out. A worker owned by a user is only offered that user's projects. -A claimed job carries everything needed to run it: image, script, -environment, services, artifact patterns, timeout, where to fetch the -source, and absolute URLs for the calls above. +`/api/v1/tasks/claim` is the only URL a worker needs to know. A claimed +task carries absolute URLs for every other call in `endpoints`, so the rest +of the surface is discovered rather than assembled: + +```json +{ + "task": { + "id": "m2y5t7deprrj9s6jvv4d", + "name": "build:arch=x86_64", + "image": "debian:bookworm-slim", + "script": ["./build.sh $ARCH"], + "env": { "ARCH": "x86_64", "CONDUCTOR_PROJECT": "demo" }, + "requires": [], "services": [], "artifacts": { "paths": ["dist/**"] }, + "workdir": "/work", "timeout": 3600, "attempt": 1, + "sha": "<40 hex>", "ref": "refs/heads/main", + "masked": [], "heartbeat_interval": 30, + "endpoints": { + "source": "https://ci.example.com/api/v1/tasks/<id>/source.tar.gz", + "log": "https://ci.example.com/api/v1/tasks/<id>/log", + "artifact": "https://ci.example.com/api/v1/tasks/<id>/artifacts", + "heartbeat": "https://ci.example.com/api/v1/tasks/<id>/heartbeat", + "done": "https://ci.example.com/api/v1/tasks/<id>/complete" + } + } +} +``` + +There is no `project_id` and no `job_id` in that payload, deliberately. The +identifiers a build legitimately wants are in `env`, where they are opaque +strings the worker copies into the container without reading. Log appends send `Content-Type: application/octet-stream` with `X-Log-Offset` set to where the worker believes it is writing. An @@ -82,17 +180,17 @@ that does not match its declared length is rejected. ``` `complete` reports the outcome. A failure with attempts remaining requeues -the job; otherwise everything downstream of it is skipped. +the task; otherwise everything downstream of it is skipped. Artifacts --------- ``` -GET /api/v1/projects/:project/runs/:run/jobs/:job/artifacts/:artifact +GET /api/v1/projects/:project/jobs/:job/tasks/:task/artifacts/:artifact ``` -The one read endpoint that is not the interface, so the job page can link -straight at a build's output. The project, run and job in the path must all -agree with the artifact. A public run is downloadable by anyone; a private -one by its owner or an administrator. The response is the bytes, or a -redirect to the object store when it can presign. +The one read endpoint that exists for a browser, so the task page can link +straight at a build's output. The project, job and task in the path must +all agree with the artifact. A public job is downloadable by anyone; a +private one by its owner or an administrator. The response is the bytes, or +a redirect to the object store when it can presign. diff --git a/docs/deployment.md b/docs/deployment.md @@ -1,7 +1,7 @@ Deployment ========== -One service, one image. The conductor accepts triggers, schedules runs, +One service, one image. The conductor accepts triggers, schedules jobs, serves the worker API and renders the interface. Workers are separate and may live anywhere, including on machines you do not administer. @@ -13,7 +13,7 @@ docker compose -f deploy/docker-compose.yml up -d ``` That pulls the published images and brings up a conductor on port 8080 -with sqlite on a volume, and one worker. The compose file needs a worker token, so the first run is +with sqlite on a volume, and one worker. The compose file needs a worker token, so the first job is two steps: ```sh @@ -78,7 +78,7 @@ Everything the conductor keeps lives in `/data`: /data/storage artifacts and archived logs, unless S3 is configured ``` -Mount a volume there. Losing it loses run history, and the mirrors and +Mount a volume there. Losing it loses job history, and the mirrors and spool will be rebuilt. Configuration @@ -135,8 +135,8 @@ docker run -d --restart unless-stopped \ finwo/conductor-worker ``` -It keeps no state and shares no directory with the jobs it runs. A job's -tree is unpacked into the job's own container over the socket, and its +It keeps no state and shares no directory with the tasks it runs. A task's +tree is unpacked into the task's own container over the socket, and its artifacts are read back the same way, so there is no workspace to mount, no path that has to mean the same thing on both sides, and nothing to persist between restarts. @@ -178,12 +178,12 @@ Retention --------- Nothing is deleted unless this says so, and logs are what fills a disk: -written for every job, read for almost none. The defaults keep artifacts -for 30 days or the last 10 runs, and logs for 14 days. +written for every task, read for almost none. The defaults keep artifacts +for 30 days or the last 10 jobs, and logs for 14 days. ```yaml retention: - artifact_keep_runs: 10 + artifact_keep_jobs: 10 artifact_keep_days: 30 log_keep_days: 14 sweep_interval: 3600 @@ -198,7 +198,7 @@ Three values, three meanings, and the difference matters: | ------ | ---------------------------------------- | | `null` | follow the server default | | `0` | keep forever | -| `n` | keep that many runs, or that many days | +| `n` | keep that many jobs, or that many days | The settings page shows what a `null` resolves to, since the server defaults are not otherwise visible there. @@ -206,22 +206,22 @@ are not otherwise visible there. ### How the artifact rules combine An artifact is kept if **either** rule wants it. The last -`artifact_keep_runs` runs keep their artifacts however old they are, and -anything younger than `artifact_keep_days` is kept however many runs have +`artifact_keep_jobs` jobs keep their artifacts however old they are, and +anything younger than `artifact_keep_days` is kept however many jobs have followed. Setting both is therefore more generous than setting one, not less. -The most recent **successful** run is kept regardless, so a project that +The most recent **successful** job is kept regardless, so a project that has not built in months still has something to download. Only the latest one: older successes are not protected. -A job that sets `artifacts.expire` in its pipeline overrides all of the +A task that sets `artifacts.expire` in its pipeline overrides all of the above with an exact deadline, including the protection for the last good -run. That is deliberate, and is how a bulky intermediate avoids becoming +job. That is deliberate, and is how a bulky intermediate avoids becoming immortal by being green. -Logs go by age alone. Sweeping a log deletes the bytes and leaves the job, -so a run stays explainable after its output is gone, and the interface can +Logs go by age alone. Sweeping a log deletes the bytes and leaves the task, +so a job stays explainable after its output is gone, and the interface can tell an expired log apart from one that was never written. ### What a first sweep will do @@ -275,7 +275,7 @@ CONDUCTOR_WORKER_IMAGE=finwo/conductor-worker:1.2.3 \ ``` Workers can be upgraded independently and in any order. The worker API is -the boundary between them, and a worker that goes away mid-job has that job +the boundary between them, and a worker that goes away mid-task has that task requeued after `scheduler.heartbeat_timeout`. Publishing @@ -295,12 +295,12 @@ A tag that is not a plain version, `v1.2.3-rc1` for instance, is refused rather than guessed at. Pushing needs `REGISTRY_USERNAME` and `REGISTRY_TOKEN` as project -variables. They are masked in job logs, and passed to `docker login` on +variables. They are masked in task logs, and passed to `docker login` on standard input rather than on the command line so they stay out of the -process list. The job logs out again on the way out, whether or not it +process list. The task logs out again on the way out, whether or not it succeeded, so nothing is left behind on a shared worker. -The foreign architectures are emulated with QEMU, registered per run with +The foreign architectures are emulated with QEMU, registered per job with `tonistiigi/binfmt`. Giving the pool a worker of that architecture makes it native instead, since a worker declares its own architectures when it polls. diff --git a/docs/pipeline.md b/docs/pipeline.md @@ -11,22 +11,31 @@ file are reported together. Top level --------- -| Key | Required | Description | -| ------------ | -------- | -------------------------------------------- | -| `version` | yes | Must be `1`. | -| `visibility` | no | `public` or `private`. Overrides the project. | -| `workdir` | no | Absolute path jobs run in. Overrides the project. | -| `defaults` | no | Values inherited by every job. | -| `jobs` | yes | Mapping of job name to job. | - -Job names may contain letters, digits, underscore, dot and hyphen, and must +| Key | Required | Description | +| ------------ | -------- | --------------------------------------------------- | +| `version` | yes | Must be `1`. | +| `visibility` | no | `public` or `private`. Overrides the project. | +| `workdir` | no | Absolute path tasks run in. Overrides the project. | +| `defaults` | no | Values inherited by every task. | +| `tasks` | yes | Mapping of task name to task. | + +The file may be named `.conductor.yml` or `.conductor.yaml`; both are +looked for unless the project was pointed at some other path explicitly. +Carrying both at once is an error rather than a coin toss, since two +pipelines in one repository will disagree sooner or later. + +This key was called `jobs` before 0.3.0. A job is now the pipeline run +that these tasks belong to, and a file still using the old spelling is +refused with a message saying so. + +Task names may contain letters, digits, underscore, dot and hyphen, and must start with a letter or digit. ```yaml version: 1 defaults: image: debian:bookworm-slim -jobs: +tasks: test: script: [make test] ``` @@ -34,7 +43,7 @@ jobs: Visibility ---------- -Runs are private unless something says otherwise, and the repository has the +Jobs are private unless something says otherwise, and the repository has the final word on its own results: ```yaml @@ -42,14 +51,14 @@ version: 1 visibility: public ``` -Unset means the project's setting stands. The value is recorded on each run -as it is created, so turning a repository private stops exposing future runs +Unset means the project's setting stands. The value is recorded on each job +as it is created, so turning a repository private stops exposing future jobs without retroactively hiding, or revealing, earlier ones. -A public run, its logs and its artifacts are readable by anyone. A private -run is readable by the project's owner and by administrators. +A public job, its logs and its artifacts are readable by anyone. A private +job is readable by the project's owner and by administrators. -Jobs +Tasks ---- | Key | Type | Default | Inheritable | @@ -73,7 +82,7 @@ Jobs ### script -A list of shell commands, run in order in the job's container. The job fails +A list of shell commands, run in order in the task's container. The task fails on the first command that exits non-zero. Take care with YAML scalars. Unquoted `true`, `false`, `yes`, `no` and `on` @@ -87,7 +96,7 @@ script: [true] # rejected, this is a boolean ### timeout Accepts whole seconds, or a duration with units `s`, `m`, `h`, `d` and `w`, -optionally combined: `90`, `90s`, `30m`, `1h`, `1h30m`. A job timeout may not +optionally combined: `90`, `90s`, `30m`, `1h`, `1h30m`. A task timeout may not exceed seven days; `artifacts.expire` may not exceed a year. Fanning out @@ -95,7 +104,7 @@ Fanning out ### only -Restricts a job to certain refs. Without it a job runs on every push. +Restricts a task to certain refs. Without it a task runs on every push. ```yaml publish: @@ -110,13 +119,13 @@ Patterns match the whole ref, so `refs/heads/main` rather than `main`. A `*` stands for any run of characters; everything else is literal, so a dot is a dot rather than any character. -A job that does not match is left out of the run entirely, not recorded -as skipped. A skipped job means a dependency collapsed and fails the run, +A task that does not match is left out of the job entirely, not recorded +as skipped. A skipped task means a dependency collapsed and fails the job, which is the wrong reading for a publish step that was never meant to run on this branch. -Anything depending on an excluded job is excluded with it. Dropping the -dependency instead would let a job run without something it declared it +Anything depending on an excluded task is excluded with it. Dropping the +dependency instead would let a task run without something it declared it needed, which is the worse surprise: ```yaml @@ -129,15 +138,15 @@ announce: script: ['make announce'] ``` -A run triggered without a ref matches no pattern, so restricted jobs stay +A job triggered without a ref matches no pattern, so restricted tasks stay out rather than being handed a ref they were never written for. The rule is not inheritable from `defaults`. A restriction that silently -applied to every job is hard to spot when the symptom is an empty run. +applied to every task is hard to spot when the symptom is an empty job. ### arch -Expands the job into one job per architecture, and restricts each to a worker +Expands the task into one task per architecture, and restricts each to a worker that offers it. The value is available to the script as `$ARCH`. ```yaml @@ -161,16 +170,16 @@ package: mode: [debug, release] ``` -produces four jobs, named `package:pkg=musl,mode=debug` and so on. +produces four tasks, named `package:pkg=musl,mode=debug` and so on. -Matrix values become part of the job name, so they are limited to letters, +Matrix values become part of the task name, so they are limited to letters, digits, underscore, dot and hyphen. Use `arch` rather than a matrix dimension called `arch`. Dependencies ------------ -`needs` lists jobs that must succeed first. A cycle is rejected, naming the +`needs` lists tasks that must succeed first. A cycle is rejected, naming the cycle. When a dependency shares a dimension with the dependent, it is matched on @@ -198,9 +207,9 @@ Three forms are accepted: ```yaml needs: [build] # match shared dimensions -needs: ['build:arch=x86_64'] # one specific job +needs: ['build:arch=x86_64'] # one specific task needs: - - job: build + - task: build match: all # every instance, ignore dimensions ``` @@ -215,7 +224,7 @@ Worker features --------------- `requires` names capabilities the worker must advertise. The conductor only -offers a job to a worker that advertises all of them. +offers a task to a worker that advertises all of them. ```yaml package: @@ -225,13 +234,13 @@ package: What a feature provides is defined on the worker, not here: a mount, some environment, a privileged container. This is how a signing key reaches a -build without the conductor ever holding it, and how a job asks for docker in +build without the conductor ever holding it, and how a task asks for docker in docker. Services -------- -Sidecar containers started alongside the job on the same network, reachable +Sidecar containers started alongside the task on the same network, reachable by their alias. ```yaml @@ -247,7 +256,7 @@ test: ``` The alias defaults to the image name without registry, path or tag, so -`postgres:16` is reachable as `postgres`. Aliases must be unique within a job. +`postgres:16` is reachable as `postgres`. Aliases must be unique within a task. Artifacts --------- @@ -265,11 +274,11 @@ A bare list is shorthand for `paths`. Paths are relative to the workspace; absolute paths are rejected. `when` is `on_success`, `on_failure` or `always`. -`expire` sets an exact deadline for what this job produces, and overrides +`expire` sets an exact deadline for what this task produces, and overrides the project's retention policy in both directions: sooner than the project would delete, or later than it would keep. It also overrides the rule that -protects the most recent successful run, which is the point of it. A -node_modules tree is worth dropping within the hour whether or not its run +protects the most recent successful job, which is the point of it. A +node_modules tree is worth dropping within the hour whether or not its job was green, and without that the largest artifacts are the ones kept longest. @@ -291,7 +300,7 @@ version: 1 workdir: /usr/src/app ``` -Where the tree is unpacked inside the job container, and the directory +Where the tree is unpacked inside the task container, and the directory every script starts in. It must be an absolute path of ordinary directory names; it does not have to exist in the image, and is created as the tree is unpacked. @@ -303,25 +312,25 @@ knows that, and whoever registered the project should not have to. Nothing else is shared with the container. The tree arrives over the docker API and artifacts leave the same way, so a worker needs no -directory in common with the jobs it runs, and no storage of its own. +directory in common with the tasks it jobs, and no storage of its own. Environment ----------- `env` values must be scalars, and names must be valid environment variable -names. Job `env` is merged over `defaults.env`. +names. Task `env` is merged over `defaults.env`. Injected automatically: | Variable | Present when | | ----------------- | -------------------------------- | -| `ARCH` | the job declares `arch` | +| `ARCH` | the task declares `arch` | | `MATRIX_<NAME>` | for each matrix dimension | Interpolation ------------- -`${{ ... }}` is substituted before the job is stored, in `image`, `script`, +`${{ ... }}` is substituted before the task is stored, in `image`, `script`, `env` values, `requires` and `services`. Two expressions are available: `arch` and `matrix.<name>`. @@ -334,7 +343,7 @@ build: script: ['./build.sh ${{ matrix.pkg }}'] ``` -Referring to a dimension the job does not declare is an error. Inside +Referring to a dimension the task does not declare is an error. Inside `script` the `$ARCH` and `$MATRIX_*` variables usually read better, since the shell expands them; interpolation exists for the fields the shell never sees, such as `image`. @@ -349,7 +358,7 @@ defaults: image: debian:bookworm-slim timeout: 30m -jobs: +tasks: lint: script: [make lint] @@ -376,6 +385,6 @@ jobs: script: ['./mk/publish.sh'] ``` -This yields nine jobs: one `lint`, two `build`, four `package` and one -`publish`. The two `build` jobs run at the same time as `lint`; each +This yields nine tasks: one `lint`, two `build`, four `package` and one +`publish`. The two `build` tasks run at the same time as `lint`; each `package` starts as soon as its own architecture's `build` finishes. diff --git a/docs/worker.md b/docs/worker.md @@ -1,10 +1,10 @@ Running a worker ================ -A worker polls a conductor for jobs and runs them in containers. It needs no +A worker polls a conductor for tasks and runs them in containers. It needs no inbound connectivity, so it can sit behind NAT, and it needs no repository credentials, because the conductor serves the source for the exact commit a -job was handed. +task was handed. The worker has no npm dependencies and keeps no state. Copying `src/worker` onto a host with node and a container runtime is enough. @@ -15,7 +15,7 @@ Requirements - node 24 or newer - a docker compatible CLI: docker, podman or nerdctl -That is the whole list. A job's tree is unpacked into the job's own +That is the whole list. A task's tree is unpacked into the task's own container over the docker API, so the worker needs no `tar` of its own, and it never touches a repository, so it needs no `git` either. @@ -61,17 +61,17 @@ node src/worker/agent.js --config worker.json | Key | Default | Description | | ---------------- | -------------------- | ----------------------------------------------- | | `conductor_url` | `http://127.0.0.1:8080` | Base URL of the conductor. | -| `name` | hostname | Shown against jobs this worker ran. | +| `name` | hostname | Shown against tasks this worker ran. | | `token` | none | The worker token, inline. | | `token_file` | none | Path to a file holding the token. Preferred. | -| `arches` | `[]` | Architectures offered. Empty means no `arch` jobs, only jobs that declare none. | +| `arches` | `[]` | Architectures offered. Empty means no `arch` tasks, only tasks that declare none. | | `features` | `{}` | Capabilities offered, see below. | -| `concurrency` | `1` | Jobs run at once. | +| `concurrency` | `1` | Tasks run at once. | | `poll_interval` | `5` | Seconds between polls when idle. | | `docker` | `docker` | Runtime CLI. Set to `podman` or `nerdctl`. | -| `shell` | `sh` | Shell used to run a job script in its image. | +| `shell` | `sh` | Shell used to run a task script in its image. | -There is nothing here about where jobs run or what they are given, which +There is nothing here about where tasks run or what they are given, which is deliberate: that is a property of the build and is decided by the pipeline or the project, not by the machine that happens to run it. @@ -84,9 +84,9 @@ Features -------- A feature is a name this worker advertises, together with whatever local -resources a job asking for it should receive. The conductor only ever learns -the name. It will not send a job to a worker that does not advertise every -feature the job's `requires` lists. +resources a task asking for it should receive. The conductor only ever learns +the name. It will not send a task to a worker that does not advertise every +feature the task's `requires` lists. ```json "features": { @@ -108,34 +108,35 @@ package: ``` This is how a signing key reaches a build without the conductor ever holding -it. The key stays on the machine that owns it, and only jobs that explicitly +it. The key stays on the machine that owns it, and only tasks that explicitly ask for it are ever scheduled there. `privileged: true` is what a docker in docker service needs. Grant it only to features you intend to be privileged, since it removes the isolation -between the job and the host. +between the task and the host. -What a job gets +What a task gets --------------- -Each job runs in its own container, on its own network: +Each task runs in its own container, on its own network: - - the repository tree at the job's commit, unpacked into the working + - the repository tree at the task's commit, unpacked into the working directory, which is `/work` unless the pipeline or the project says otherwise - that directory as the working directory - - the job's `env`, plus `ARCH` and `MATRIX_*` for a fanned out job - - `CONDUCTOR_PROJECT`, `CONDUCTOR_RUN_ID`, `CONDUCTOR_RUN_NUMBER`, - `CONDUCTOR_JOB`, `CONDUCTOR_JOB_ID`, `CONDUCTOR_SHA`, `CONDUCTOR_REF` - and `CONDUCTOR_ATTEMPT` - - any `services`, reachable by their alias on the job network + - the task's `env`, plus `ARCH` and `MATRIX_*` for a fanned out task + - `CONDUCTOR_PROJECT`, `CONDUCTOR_JOB_ID`, `CONDUCTOR_JOB_NUMBER`, + `CONDUCTOR_TASK`, `CONDUCTOR_TASK_ID`, `CONDUCTOR_SHA`, `CONDUCTOR_REF` + and `CONDUCTOR_ATTEMPT`, which name the project and job the worker + itself is never told about: they are opaque strings it copies in + - any `services`, reachable by their alias on the task network - mounts and environment from the features it requires Output is streamed to the conductor as it happens. Artifacts are read -back out of the container when the job finishes, and uploaded. +back out of the container when the task finishes, and uploaded. Nothing else is shared. The worker mounts no directory of its own into a -job: the tree goes in over the docker API and the artifacts come back the +task: the tree goes in over the docker API and the artifacts come back the same way. That is why a worker needs no volumes, no matching paths between itself and the host, and no storage at all. @@ -143,20 +144,20 @@ Operational notes ----------------- **Cleanup.** Containers and networks are removed whatever the outcome. -There is no workspace to clean up, and so no trouble with a job having +There is no workspace to clean up, and so no trouble with a task having written files as root that the worker then cannot delete: everything the -job wrote lived in its container and went with it. +task wrote lived in its container and went with it. **Shutdown.** On SIGINT or SIGTERM the worker stops polling and lets running -jobs finish. A second signal exits immediately, and the conductor will +tasks finish. A second signal exits immediately, and the conductor will eventually reap whatever was left running. -**Failures.** A worker that stops reporting has its jobs requeued or failed +**Failures.** A worker that stops reporting has its tasks requeued or failed by the conductor after `scheduler.heartbeat_timeout`. Losing a worker never -strands a run. +strands a job. -**Log loss.** If the conductor cannot be reached while a job runs, log output -for that window is dropped rather than failing the job. The job's outcome is +**Log loss.** If the conductor cannot be reached while a task runs, log output +for that window is dropped rather than failing the task. The task's outcome is still reported. Trust @@ -165,9 +166,9 @@ Trust A worker is trusted with the source of the commits it builds and with whatever its features grant it. It is not trusted with anything else: it holds no repository credentials, it cannot read other refs, it cannot reach -another project's jobs, and it cannot write to storage except by uploading -artifacts for a job it currently holds. +another project's tasks, and it cannot write to storage except by uploading +artifacts for a task it currently holds. -Running a job means running code from the repository. Only accept work from +Running a task means running code from the repository. Only accept work from a conductor whose projects you are willing to execute, and keep features narrow. diff --git a/examples/distro.conductor.yml b/examples/distro.conductor.yml @@ -20,7 +20,7 @@ defaults: image: debian:bookworm-slim timeout: 2h -jobs: +tasks: check: timeout: 10m script: diff --git a/examples/node-app.conductor.yml b/examples/node-app.conductor.yml @@ -9,7 +9,7 @@ defaults: image: node:22-bookworm-slim timeout: 20m -jobs: +tasks: install: script: - npm ci