conductor

CI task system
git clone git://git.finwo.net/app/conductor
Log | Files | Refs | README | LICENSE

conductor-worker.md (4626B)

      1 # conductor-worker
      2 
      3 Runs tasks for a [conductor](https://hub.docker.com/r/finwo/conductor)
      4 server. Polls for work, runs each task in its own container, streams the
      5 log back and uploads artifacts.
      6 
      7 Source and issues: https://github.com/finwo/conductor
      8 
      9 ## Tags
     10 
     11 | Tag              | What it is                                 |
     12 | ---------------- | ------------------------------------------ |
     13 | `latest`         | the most recent release                    |
     14 | `0.4.0`          | an exact version                           |
     15 | `0.4`            | the latest patch of that minor series      |
     16 | `<commit>`       | an exact build from main, twelve hex chars |
     17 
     18 Built for `linux/amd64`, `linux/arm64` and `linux/riscv64`.
     19 
     20 **Upgrade the conductor and its workers together.** In 0.3.0 a job became
     21 one run of a pipeline and a task became the unit this image runs. Workers
     22 claim from `/api/v1/tasks/claim`, so a 0.2.x worker will not run anything
     23 for a 0.3 or newer conductor. Pin both to the same version if they are
     24 upgraded separately.
     25 
     26 ## Quick start
     27 
     28 A worker needs no inbound connectivity, so this works from behind NAT.
     29 Mint a token in the interface under Workers, or with the admin CLI on the
     30 server, then:
     31 
     32 ```sh
     33 docker run -d \
     34   --name conductor-worker \
     35   --restart unless-stopped \
     36   -e CONDUCTOR_URL=https://ci.example.com \
     37   -e CONDUCTOR_WORKER_TOKEN=... \
     38   -v /var/run/docker.sock:/var/run/docker.sock \
     39   finwo/conductor-worker
     40 ```
     41 
     42 A worker you register is only ever offered tasks from your own projects.
     43 
     44 ## It keeps nothing
     45 
     46 The socket is the only thing this image needs. A task's tree is unpacked
     47 into the task's own container over that socket, and its artifacts are read
     48 back the same way, so the worker shares no directory with what it runs.
     49 
     50 That means no volumes, nothing to persist between restarts, and no paths
     51 that have to mean the same thing inside and outside the container. The
     52 task containers are siblings on the host daemon rather than children, but
     53 since nothing of the worker's filesystem is handed to them, that stops
     54 being something you have to think about.
     55 
     56 Where a task runs is decided by the build, not by this machine: the
     57 `workdir` key in the pipeline, the project's setting, or `/work`.
     58 
     59 ## Configuration
     60 
     61 | Variable                       | Default        | What it does                        |
     62 | ------------------------------ | -------------- | ----------------------------------- |
     63 | `CONDUCTOR_URL`                | required       | Where the conductor is.             |
     64 | `CONDUCTOR_WORKER_TOKEN`       | required       | Issued by the conductor, shown once. |
     65 | `CONDUCTOR_WORKER_NAME`        | the hostname   | How this worker appears in tasks.    |
     66 | `CONDUCTOR_WORKER_CONCURRENCY` | `1`            | Tasks at once.                       |
     67 | `CONDUCTOR_WORKER_ARCHES`      | any            | Architectures offered, comma separated. |
     68 | `CONDUCTOR_WORKER_TOKEN_FILE`  | none           | Read the token from a file instead. |
     69 | `CONDUCTOR_WORKER_DOCKER`      | `docker`       | Runtime CLI: docker, podman, nerdctl. |
     70 | `CONDUCTOR_WORKER_POLL_INTERVAL` | `5`          | Seconds between polls when idle.    |
     71 
     72 Leaving `CONDUCTOR_WORKER_ARCHES` unset offers every architecture, which
     73 is what you want unless one worker serves several.
     74 
     75 ## Features
     76 
     77 A feature is a name this worker advertises, together with the local
     78 resources a task asking for it should be given. A pipeline task asks with
     79 `requires:`, and the conductor will not send that task to a worker which
     80 does not advertise every name it lists. The conductor only ever learns
     81 the names.
     82 
     83 Features cannot be expressed in an environment variable, because each one
     84 carries mounts, devices and environment of its own. They need a
     85 configuration file, JSON or YAML, mounted into the container:
     86 
     87 ```json
     88 {
     89   "conductor_url": "https://ci.example.com",
     90   "features": {
     91     "docker": {},
     92     "sign-key": {
     93       "mounts": ["/srv/keys/build.rsa:/keys/build.rsa:ro"],
     94       "env": { "SIGN_KEY": "/keys/build.rsa" }
     95     }
     96   }
     97 }
     98 ```
     99 
    100 ```sh
    101 -v /etc/conductor/worker.json:/etc/conductor/worker.json:ro
    102 ```
    103 
    104 A file at that path is picked up on its own, with the token still coming
    105 from the environment if you would rather keep it out of the file.
    106 Nothing enforces what a name means beyond matching it, so `docker` is
    107 simply how you say this worker's tasks may use the socket mounted above.
    108 
    109 ## Documentation
    110 
    111 - [Workers](https://github.com/finwo/conductor/blob/main/docs/worker.md)
    112 - [Pipelines](https://github.com/finwo/conductor/blob/main/docs/pipeline.md)
    113 - [Deployment](https://github.com/finwo/conductor/blob/main/docs/deployment.md)