unos-repository

README.md (12193B)

---

 # mk/
 
 UNOS package driver: `channels/<layer>/packages/<name>/template` in, signed
 `.apk` (v2) out. The whole of `build/` is generated output.
 
 ## Channels
 
 Packages live in layers under `channels/`. Each channel declares its parent in
 `channel.conf` (`upstream=`), and composing a channel means applying every
 layer from the root down, replacing whole package directories. `publish=no`
 marks a layer that is never built in CI and never uploaded.
 
     base              publish=no, the foundation
     lts               upstream=base
     stable            upstream=lts
     edge              upstream=stable
 
 `channel-assemble.sh <channel>` writes the composed tree to
 `build/channels/<channel>/packages`, which is what every other script reads.
 Published at `apk/<channel>/<arch>/`; there is no `base` in the bucket.
 
 Promoting a package between channels is a `git mv`, never a copy --
 `channel-lint.sh` fails the build on a layer identical to the one it shadows,
 because such a layer silently stops tracking its upstream forever.
 
 ## Builder image
 
 Pipeline tasks run on `finwo/unos-builder:<builder/VERSION>` rather than
 installing an apt toolchain per task. `builder/Dockerfile` bakes it by running
 `mk/ci-setup.sh --install-only`, so the image and the toolchain list cannot
 drift apart.
 
 Tags are immutable: `publish-builder.sh` leaves an existing tag alone and
 reports the task skipped. A toolchain change therefore needs a `builder/VERSION`
 bump to take effect, and forgetting that bump is caught by `ci-setup.sh --check`
 in the `check` task, which runs inside the image that was pushed.
 
 Needs the `docker` worker feature and `REGISTRY_USERNAME` / `REGISTRY_TOKEN`
 project variables (`REGISTRY_HOST` for a registry other than Docker Hub).
 
 ## Files
 
 - `publish-builder.sh [--dry-run]` -- build and push the builder image.
 - `channels.inc` -- chain resolution and repo path layout. Sourced, not run.
 - `channel-assemble.sh <channel>` -- compose a channel's package tree.
 - `channel-lint.sh` -- reject dead overlays.
 - `bootstrap-host.sh [--check|--force]` -- builds the host tools in
   `build/host/bin` from pinned, checksummed tarballs: `muon` (bootstrapped from
   a single amalgamated C file, so the chain terminates at a C compiler) and
   `apk`. `--check` verifies presence *and version* and is the gate CI should
   run first. Everything else assumes these exist.
 - `deps.sh [--channel <c>] {list|vars|graph|order|rdeps|changed|check|seed|owner|
   subpkgs|pkgver|rtdeps|channels|layers}` -- the build-dependency graph, read
   from each template's `makedepends`. `order` is a topological sort; `check`
   validates it (dangling deps, self-deps, cycles, missing hostmakedepends).
   Defaults to `$UNOS_CHANNEL`, then `lts`.
 - `gen-pipeline.sh [--write|--check]` -- emits `.conductor.yml` from that
   graph. The pipeline is data (the conductor will not run repository code to
   build its task graph), so the per-package `needs` edges must be written out
   literally; generating them keeps `makedepends` the single source of truth.
   `--check` is run by `tests/unit/test_deps.sh`, so a template change that is
   not reflected in the pipeline fails the suite.
 - `ci-setup.sh [--check]` -- installs the build toolchain into an ephemeral CI
   container, then verifies every `hostmakedepends` in the graph resolves. That
   verification is the point: it stops the apt list falling behind a template.
 - `sysroot.sh` -- seed `build/sysroot` from the published apk repo, so a CI
   container can build one package without rebuilding the other 26.
 - `publish.sh [--arch A] [--index-only] [pkg...]` -- upload `.apk` files and
   `APKINDEX.tar.gz` to the S3 bucket serving the public repo, using
   `curl --aws-sigv4` (no awscli or rclone). Reads `BUCKET_ACCESS_KEY`,
   `BUCKET_SECRET_KEY`, `BUCKET_NAME`, `BUCKET_ENDPOINT` (URL) and
   `BUCKET_REGION` (region name); it rejects those last two transposed, since
   the natural symptom is an opaque signature mismatch. The index is always
   uploaded last, so it never advertises packages that are not there yet.
 - `build-all.sh [--dry-run] [--skip-built] [pkg...]` -- build everything, or
   one package and its closure, in `deps.sh order`.
 - `build.sh <pkg>` -- the pipeline: source template -> check
   `hostmakedepends`/`makedepends` -> fetch+verify -> extract -> patches ->
   `do_build`/`do_install` into `DESTDIR` -> emit signed `.apk` into
   `build/repo/<arch>/`. Dependency checks run before any fetching, so a missing
   host tool is named up front rather than surfacing 200 lines into someone
   else's generated parser.
 - `repo-index.sh [arch]` -- rebuild `APKINDEX.tar.gz` with host `apk index`,
   then sign it in place (abuild-sign layout: gzipped sig tar record
   prepended, RSA/SHA1 over the index bytes).
 - `pax-tar.c` -- tar writer for all three segments (C, std=c99, no
   dependencies): sorted entries, uid/gid 0, per-file SHA1 in
   `APK-TOOLS.checksum.SHA1` pax headers in data-segment mode (`--no-checksum`
   for control/signature). Compiled on first use by the scripts; the binary is
   output, not source. Host `tar` is used only for extracting upstreams.
 - `sign-key.inc` -- `resolve_sign_key()` sourced by both scripts above.
   Precedence: `$UNOS_SIGN_KEY` > repo-root `.sign-key` > single key in
   `~/.unos-keys/`.
 - `keymgmt.sh` -- key management: `new [--type rsa|supercop] [--for apk|unos]
   <stem>` generates `<stem>-<8 hex>` in `~/.unos-keys/` (rsa: 4096-bit;
   supercop: ed25519) and stages the public half into
   `channels/base/packages/unos-keys/files/<for>/`; `use <name|file>` pins the package
   signing key by writing its full path to `.sign-key` (never committed; CI
   sets `UNOS_SIGN_KEY` or writes `.sign-key` itself); `list` shows local keys
   and marks the active one. apk mandates RSA -- supercop-for-apk is refused.
 
 C and shell only. Python is banned from our tooling and from every rootfs;
 the sole exception is an upstream package whose own build mandates it
 (glibc needs python3 to build) - used for that build only, never shipped.
 
 ## Template contract
 
 Shell fragment, not a script. Variables: `pkgname`, `version`, `revision`
 (default 0), `short_desc`, `maintainer`, `license`, `homepage`, `distfiles`
 (space-separated URLs, empty = no upstream source), `checksum`
 (space-separated sha256, same order -- mandatory when `distfiles` is set),
 `depends`, `provides`, `replaces` (space-separated, may be empty).
 
 Optional phase overrides: `do_configure`, `do_build`, `do_install` (default:
 no-op). Environment and helpers provided: `DESTDIR`, `WRKSRC`, `FILESDIR`, `WORK`
 (scratch root for out-of-tree builds), `vinstall <file> <mode> <targetdir>
 [name]`, `vmkdir <dir>`, `msg`, `die`.
 
 Control-script pickup: `pre-install`, `post-install`, `pre-deinstall`,
 `post-deinstall`, `pre-upgrade`, `post-upgrade`, `trigger` in `files/` are
 packed as `.<name>` into the control segment when present.
 
 ## Keys
 
 `UNOS_SIGN_KEY` selects the RSA key (4096-bit minimum); unset means the
 single `~/.unos-keys/*.rsa` key. Keys never enter git. The public half
 (`<name>.rsa.pub`) is what lands in `/etc/apk/keys` via the `unos-keys`
 package; v2 requires the on-disk filename to equal the key name embedded in
 `.SIGN.RSA.<name>.rsa.pub`, which the driver guarantees by construction.
 
 Key ceremony: private halves live only in `~/.unos-keys/`. Public halves are
 committed by `keymgmt.sh new` under `channels/base/packages/unos-keys/files/<for>/` (public
 by design) and installed by that package -- `files/apk/*` to `/etc/apk/keys/`,
 `files/unos/*` to `/etc/unos/keys/`. Rotation = generate, commit the new
 `.pub` files, bump the `unos-keys` version.
 
 ## Repo layout
 
 `build/repo/<arch>/` holds `*.apk` + signed `APKINDEX.tar.gz`. The arch level
 is mandatory: apk appends `$arch` to the repository URL when fetching the
 index. Current keys (dev estate, rotate before any release):
 
 - RSA/apk: `unos-dev@finwo.dev-096b7b41`
 - ed25519/supercop: `unos-dev@finwo.dev-6a2764d4`
 
 ## Format notes (learned against real Alpine packages)
 
 - `.apk` v2 = signed-control gzip member + data gzip member. The signature
   covers the control gzip bytes; the control tar carries NO end-of-tar
   markers (only the data segment terminates the archive).
 - `datahash` is plain sha256 hex of `data.tar.gz` -- no `Q1` prefix (that
   prefix form lives in indexes, where `apk index` computes it itself).
 - Per-file `APK-TOOLS.checksum.SHA1` records are required for symlinks too
   (v2 warns otherwise): symlinks hash the target string, regular files their
   content (verified against `abuild-tar.c`).
 
 ## Host dependencies
 
 sh, cc, curl or wget, tar, gzip, openssl, sha256sum, install, patch.
 apk-tools and muon are **not** host prerequisites -- `bootstrap-host.sh` builds
 them into `build/host/bin` from pinned sources.
 
 Per-package host tools (bison, flex, bc, perl, python3, ...) are declared as
 `hostmakedepends` in the templates that need them and checked by `build.sh`, so
 a CI image can be validated against the real requirement set with:
 
 ```sh
 ./mk/deps.sh check
 ```
 
 A clean build from nothing is therefore:
 
 ```sh
 ./mk/bootstrap-host.sh     # muon + apk into build/host/bin
 ./mk/build-all.sh          # every package, in dependency order
 ./mk/repo-index.sh         # signed APKINDEX
 ./mk/publish.sh            # upload packages + index to S3
 ```
 
 ## Assembling an OS
 
 Not here. `rootfs.sh`, `chroot.sh`, `make-img.sh`, `run-qemu.sh` and the
 installer live in the `unos` repo, which consumes the published repository over
 HTTP rather than reaching into `build/repo` on a shared filesystem. This repo
 produces packages and stops there.
 
 ## Host apk-tools v3 (bootstrap input, pinned)
 
 Upstream `apk-tools` v3.0.8, tarball
 `https://gitlab.alpinelinux.org/alpine/apk-tools/-/archive/v3.0.8/apk-tools-v3.0.8.tar.gz`
 sha256 `e81c64a6e7c3806d45d4622c253e757aee970be84048554fa0eed79310e9453e`,
 fetched once into `build/work/sources/`. Built in-tree **with muon, never
 meson** (Python is banned from our tooling; upstream supports muon):
 
 - dynamic, prefix `build/host/` (`-Dlua=disabled -Dpython=disabled
   -Dhelp=disabled -Ddocs=disabled`, rpath `$ORIGIN/../lib`):
   `build/host/bin/apk`, used by `repo-index.sh` for index generation. The
   `unos` repo bootstraps its own copy for rootfs assembly.
 
 There is deliberately **no `apk.static`**. An earlier revision built one and
 justified it here as "the future target binary - no libcrypto needed on the
 switch". That was already wrong: UNOS packages `openssl`, so `libcrypto.so.3`
 and `libssl.so.3` are present on the switch and the target `apk` links them
 dynamically. Its only consumer was rootfs assembly, which produces a
 byte-identical rootfs with the dynamic binary. Building it cost 10.6MB against
 2.0MB and
 required `libcrypto.a`/`libssl.a`/`libz.a`/`libzstd.a` on the build host --
 static libraries that most CI containers do not ship, turning a cosmetic
 preference into a hard bootstrap failure. If a static apk is ever wanted for
 recovering a switch with a broken loader, it belongs in `channels/` as a target
 artifact.
 
 Muon itself is pinned: v0.6.0,
 `https://github.com/muon-build/muon/archive/refs/tags/0.6.0.tar.gz`,
 sha256 `5300e58c4b4d43e3026856004c79d746075aaa9d9e66d76ba9f32ce249495b81`,
 bootstrapped (`bootstrap.sh`, pure C) into `build/host/bin/muon`. muon is
 GPL-3.0 - fine as a host-only build tool (same standing as GCC); it never
 enters any package or rootfs.
 
 Our v2 artifacts verify and install cleanly under both v2.14 (host) and
 v3.0.8, untrusted and fully trusted (signed index + seeded key). v3 needs
 `--usermode` for non-root `--root` installs. This becomes a real package
 template once the toolchain work lands; until then the pin + hash above are
 the record.
 
 **This is no longer prose to follow by hand** -- `mk/bootstrap-host.sh` is the
 executable form of the above. `--check` verifies presence *and version* (a
 stale binary from an older pin is worse than a missing one, because everything
 downstream still appears to work), and `--force` rebuilds from the pinned
 tarballs. muon's `meson-docs`/`meson-tests` subprojects are disabled because
 its `setup` step otherwise git-clones them from GitHub mid-build, which has no
 place in a bootstrap driven by checksummed tarballs.