conductor

CI task system
git clone git://git.finwo.net/app/conductor
Log | Files | Refs | README | LICENSE

README.md (4557B)

      1 conductor
      2 =========
      3 
      4 A stand-alone CI system. Pipelines live in the repository, tasks run in
      5 containers, and workers pull work rather than being pushed to.
      6 
      7 ```yaml
      8 # .conductor.yml
      9 version: 1
     10 
     11 defaults:
     12   image: debian:bookworm-slim
     13 
     14 tasks:
     15   test:
     16     script: [make test]
     17 
     18   version:
     19     script:
     20       - echo "::set-output tag=$(cat VERSION)::public"
     21 
     22   build:
     23     needs: [version]
     24     arch: [x86_64, aarch64]
     25     script:
     26       - ./build.sh $ARCH ${{ tasks.version.outputs.tag }}
     27 
     28   publish:
     29     needs: [build, test]
     30     script: [./publish.sh]
     31 ```
     32 
     33 `arch` and `matrix` fan a task out into one task per combination. A dependent
     34 sharing a dimension is matched on it; `publish` has no architecture of its
     35 own, so it waits for every build.
     36 
     37 A script hands a value to a later task with `::set-output`, read back as
     38 `${{ tasks.<task>.outputs.<name> }}`, which is itself a reason to wait for
     39 that task. Outputs are masked unless marked `::public`, so a token passed
     40 between tasks stays out of the logs.
     41 
     42 Quick start
     43 -----------
     44 
     45 ```sh
     46 docker compose -f deploy/docker-compose.yml up -d conductor
     47 ```
     48 
     49 Or from source, needing node 24 or newer and no native modules:
     50 
     51 ```sh
     52 npm install && npm start
     53 ```
     54 
     55 Either way an administrator account is created on first start and its
     56 password written to the log once. Then register a project and a worker:
     57 
     58 ```sh
     59 node src/admin-cli.js project:add demo https://git.example.com/demo.git
     60 node src/admin-cli.js token:add builder-1
     61 node src/worker/agent.js --config worker.json
     62 ```
     63 
     64 Both commands print a secret once. The trigger secret goes in the git hook
     65 (`hooks/post-receive`), the worker token in the worker configuration.
     66 GitHub, Gitea and GitLab webhooks are accepted at the same endpoint.
     67 
     68 How it works
     69 ------------
     70 
     71 The conductor reads `.conductor.yml` at the pushed commit and turns it into
     72 a graph of tasks. Workers poll for work, advertising the architectures and
     73 features they offer; any task whose dependencies have succeeded is eligible,
     74 so a wide graph runs wide.
     75 
     76 Workers need no repository credentials. The conductor serves the source tree
     77 for the exact commit a task was handed, so someone else's hardware can build
     78 for you without being able to read anything else.
     79 
     80 Projects and workers have owners. A worker you register is only ever offered
     81 tasks from your own projects, which is what makes it safe to accept build
     82 capacity from people you do not otherwise trust.
     83 
     84 Configuration
     85 -------------
     86 
     87 Nothing has to be configured; every setting has a default, so an empty file
     88 is valid. Three capabilities switch on when configured and fall back when
     89 not:
     90 
     91 | Setting                   | Configured           | Not configured          |
     92 | ------------------------- | -------------------- | ----------------------- |
     93 | `database.url`            | mysql/tidb, postgres | sqlite at a file path   |
     94 | `storage.s3.bucket`       | S3 compatible store  | local filesystem        |
     95 | `auth.oidc.discovery_url` | that provider        | the built-in login page |
     96 
     97 See `conductor.example.yaml` for the annotated list.
     98 
     99 Documentation
    100 -------------
    101 
    102 | Document                                   | Covers                                |
    103 | ------------------------------------------ | ------------------------------------- |
    104 | [pipeline.md](docs/pipeline.md)            | the `.conductor.yml` reference        |
    105 | [worker.md](docs/worker.md)                | running a worker                      |
    106 | [auth.md](docs/auth.md)                    | accounts, OIDC, who may see what      |
    107 | [deployment.md](docs/deployment.md)        | images, state, production settings    |
    108 | [api.md](docs/api.md)                      | the HTTP API                          |
    109 | [contributing.md](docs/contributing.md)    | layout, testing, building             |
    110 
    111 Status
    112 ------
    113 
    114 Under construction, and usable. Working today: sqlite, mysql/tidb and
    115 postgres; local and S3 compatible storage; pipeline parsing with matrix and
    116 architecture expansion; triggers, scheduling, live logs, artifacts and
    117 reaping; containerised tasks with services, features and caches; accounts,
    118 OIDC, ownership and project variables; a JSON API with a single page
    119 application in front of it; images for amd64, arm64 and riscv64.
    120 
    121 Testing
    122 -------
    123 
    124 ```sh
    125 npm test
    126 npm run build
    127 ```
    128 
    129 With docker present the suite starts MinIO, an OIDC provider and real task
    130 containers, and cleans up after itself. `npm run build` compiles the
    131 frontend, which the test suite does not cover. See
    132 [docs/contributing.md](docs/contributing.md).
    133 
    134 License
    135 -------
    136 
    137 See [LICENSE.md](LICENSE.md).