README.md (10194B)
1 # mk/ 2 3 UNOS package driver: `packages/<name>/template` in, signed `.apk` (v2) out. 4 The whole of `build/` is generated output. Only `mk/` is source. 5 6 ## Files 7 8 - `bootstrap-host.sh [--check|--force]` -- builds the host tools in 9 `build/host/bin` from pinned, checksummed tarballs: `muon` (bootstrapped from 10 a single amalgamated C file, so the chain terminates at a C compiler) and 11 `apk`. `--check` verifies presence *and version* and is the gate CI should 12 run first. Everything else assumes these exist. 13 - `deps.sh {list|vars|graph|order|rdeps|changed|check}` -- the build-dependency 14 graph, read from each template's `makedepends`. `order` is a topological 15 sort; `check` validates it (dangling deps, self-deps, cycles, missing 16 hostmakedepends). 17 - `gen-pipeline.sh [--write|--check]` -- emits `.conductor.yml` from that 18 graph. The pipeline is data (the conductor will not run repository code to 19 build its task graph), so the per-package `needs` edges must be written out 20 literally; generating them keeps `makedepends` the single source of truth. 21 `--check` is run by `tests/unit/test_deps.sh`, so a template change that is 22 not reflected in the pipeline fails the suite. 23 - `ci-setup.sh [--check]` -- installs the build toolchain into an ephemeral CI 24 container, then verifies every `hostmakedepends` in the graph resolves. That 25 verification is the point: it stops the apt list falling behind a template. 26 - `sysroot.sh` -- seed `build/sysroot` from the published apk repo, so a CI 27 container can build one package without rebuilding the other 26. 28 - `publish.sh [--arch A] [--index-only] [pkg...]` -- upload `.apk` files and 29 `APKINDEX.tar.gz` to the S3 bucket serving the public repo, using 30 `curl --aws-sigv4` (no awscli or rclone). Reads `BUCKET_ACCESS_KEY`, 31 `BUCKET_SECRET_KEY`, `BUCKET_NAME`, `BUCKET_ENDPOINT` (URL) and 32 `BUCKET_REGION` (region name); it rejects those last two transposed, since 33 the natural symptom is an opaque signature mismatch. The index is always 34 uploaded last, so it never advertises packages that are not there yet. 35 - `build-all.sh [--dry-run] [--skip-built] [pkg...]` -- build everything, or 36 one package and its closure, in `deps.sh order`. 37 - `build.sh <pkg>` -- the pipeline: source template -> check 38 `hostmakedepends`/`makedepends` -> fetch+verify -> extract -> patches -> 39 `do_build`/`do_install` into `DESTDIR` -> emit signed `.apk` into 40 `build/repo/<arch>/`. Dependency checks run before any fetching, so a missing 41 host tool is named up front rather than surfacing 200 lines into someone 42 else's generated parser. 43 - `repo-index.sh [arch]` -- rebuild `APKINDEX.tar.gz` with host `apk index`, 44 then sign it in place (abuild-sign layout: gzipped sig tar record 45 prepended, RSA/SHA1 over the index bytes). 46 - `pax-tar.c` -- tar writer for all three segments (C, std=c99, no 47 dependencies): sorted entries, uid/gid 0, per-file SHA1 in 48 `APK-TOOLS.checksum.SHA1` pax headers in data-segment mode (`--no-checksum` 49 for control/signature). Compiled on first use by the scripts; the binary is 50 output, not source. Host `tar` is used only for extracting upstreams. 51 - `sign-key.inc` -- `resolve_sign_key()` sourced by both scripts above. 52 Precedence: `$UNOS_SIGN_KEY` > repo-root `.sign-key` > single key in 53 `~/.unos-keys/`. 54 - `keymgmt.sh` -- key management: `new [--type rsa|supercop] [--for apk|unos] 55 <stem>` generates `<stem>-<8 hex>` in `~/.unos-keys/` (rsa: 4096-bit; 56 supercop: ed25519) and stages the public half into 57 `packages/unos-keys/files/<for>/`; `use <name|file>` pins the package 58 signing key by writing its full path to `.sign-key` (never committed; CI 59 sets `UNOS_SIGN_KEY` or writes `.sign-key` itself); `list` shows local keys 60 and marks the active one. apk mandates RSA -- supercop-for-apk is refused. 61 62 C and shell only. Python is banned from our tooling and from every rootfs; 63 the sole exception is an upstream package whose own build mandates it 64 (glibc needs python3 to build) - used for that build only, never shipped. 65 66 ## Template contract 67 68 Shell fragment, not a script. Variables: `pkgname`, `version`, `revision` 69 (default 0), `short_desc`, `maintainer`, `license`, `homepage`, `distfiles` 70 (space-separated URLs, empty = no upstream source), `checksum` 71 (space-separated sha256, same order -- mandatory when `distfiles` is set), 72 `depends`, `provides`, `replaces` (space-separated, may be empty). 73 74 Optional phase overrides: `do_configure`, `do_build`, `do_install` (default: 75 no-op). Environment and helpers provided: `DESTDIR`, `WRKSRC`, `FILESDIR`, `WORK` 76 (scratch root for out-of-tree builds), `vinstall <file> <mode> <targetdir> 77 [name]`, `vmkdir <dir>`, `msg`, `die`. 78 79 Control-script pickup: `pre-install`, `post-install`, `pre-deinstall`, 80 `post-deinstall`, `pre-upgrade`, `post-upgrade`, `trigger` in `files/` are 81 packed as `.<name>` into the control segment when present. 82 83 ## Keys 84 85 `UNOS_SIGN_KEY` selects the RSA key (4096-bit minimum); unset means the 86 single `~/.unos-keys/*.rsa` key. Keys never enter git. The public half 87 (`<name>.rsa.pub`) is what lands in `/etc/apk/keys` via the `unos-keys` 88 package; v2 requires the on-disk filename to equal the key name embedded in 89 `.SIGN.RSA.<name>.rsa.pub`, which the driver guarantees by construction. 90 91 Key ceremony: private halves live only in `~/.unos-keys/`. Public halves are 92 committed by `keymgmt.sh new` under `packages/unos-keys/files/<for>/` (public 93 by design) and installed by that package -- `files/apk/*` to `/etc/apk/keys/`, 94 `files/unos/*` to `/etc/unos/keys/`. Rotation = generate, commit the new 95 `.pub` files, bump the `unos-keys` version. 96 97 ## Repo layout 98 99 `build/repo/<arch>/` holds `*.apk` + signed `APKINDEX.tar.gz`. The arch level 100 is mandatory: apk appends `$arch` to the repository URL when fetching the 101 index. Current keys (dev estate, rotate before any release): 102 103 - RSA/apk: `unos-dev@finwo.dev-096b7b41` 104 - ed25519/supercop: `unos-dev@finwo.dev-6a2764d4` 105 106 ## Format notes (learned against real Alpine packages) 107 108 - `.apk` v2 = signed-control gzip member + data gzip member. The signature 109 covers the control gzip bytes; the control tar carries NO end-of-tar 110 markers (only the data segment terminates the archive). 111 - `datahash` is plain sha256 hex of `data.tar.gz` -- no `Q1` prefix (that 112 prefix form lives in indexes, where `apk index` computes it itself). 113 - Per-file `APK-TOOLS.checksum.SHA1` records are required for symlinks too 114 (v2 warns otherwise): symlinks hash the target string, regular files their 115 content (verified against `abuild-tar.c`). 116 117 ## Host dependencies 118 119 sh, cc, curl or wget, tar, gzip, openssl, sha256sum, install, patch. 120 apk-tools and muon are **not** host prerequisites -- `bootstrap-host.sh` builds 121 them into `build/host/bin` from pinned sources. 122 123 Per-package host tools (bison, flex, bc, perl, python3, ...) are declared as 124 `hostmakedepends` in the templates that need them and checked by `build.sh`, so 125 a CI image can be validated against the real requirement set with: 126 127 ```sh 128 ./mk/deps.sh check 129 ``` 130 131 A clean build from nothing is therefore: 132 133 ```sh 134 ./mk/bootstrap-host.sh # muon + apk into build/host/bin 135 ./mk/build-all.sh # every package, in dependency order 136 ./mk/repo-index.sh # signed APKINDEX 137 ./mk/publish.sh # upload packages + index to S3 138 ``` 139 140 ## Assembling an OS 141 142 Not here. `rootfs.sh`, `chroot.sh`, `make-img.sh`, `run-qemu.sh` and the 143 installer live in the `unos` repo, which consumes the published repository over 144 HTTP rather than reaching into `build/repo` on a shared filesystem. This repo 145 produces packages and stops there. 146 147 ## Host apk-tools v3 (bootstrap input, pinned) 148 149 Upstream `apk-tools` v3.0.8, tarball 150 `https://gitlab.alpinelinux.org/alpine/apk-tools/-/archive/v3.0.8/apk-tools-v3.0.8.tar.gz` 151 sha256 `e81c64a6e7c3806d45d4622c253e757aee970be84048554fa0eed79310e9453e`, 152 fetched once into `build/work/sources/`. Built in-tree **with muon, never 153 meson** (Python is banned from our tooling; upstream supports muon): 154 155 - dynamic, prefix `build/host/` (`-Dlua=disabled -Dpython=disabled 156 -Dhelp=disabled -Ddocs=disabled`, rpath `$ORIGIN/../lib`): 157 `build/host/bin/apk`, used by `repo-index.sh` for index generation. The 158 `unos` repo bootstraps its own copy for rootfs assembly. 159 160 There is deliberately **no `apk.static`**. An earlier revision built one and 161 justified it here as "the future target binary - no libcrypto needed on the 162 switch". That was already wrong: UNOS packages `openssl`, so `libcrypto.so.3` 163 and `libssl.so.3` are present on the switch and the target `apk` links them 164 dynamically. Its only consumer was rootfs assembly, which produces a 165 byte-identical rootfs with the dynamic binary. Building it cost 10.6MB against 166 2.0MB and 167 required `libcrypto.a`/`libssl.a`/`libz.a`/`libzstd.a` on the build host -- 168 static libraries that most CI containers do not ship, turning a cosmetic 169 preference into a hard bootstrap failure. If a static apk is ever wanted for 170 recovering a switch with a broken loader, it belongs in `packages/` as a target 171 artifact. 172 173 Muon itself is pinned: v0.6.0, 174 `https://github.com/muon-build/muon/archive/refs/tags/0.6.0.tar.gz`, 175 sha256 `5300e58c4b4d43e3026856004c79d746075aaa9d9e66d76ba9f32ce249495b81`, 176 bootstrapped (`bootstrap.sh`, pure C) into `build/host/bin/muon`. muon is 177 GPL-3.0 - fine as a host-only build tool (same standing as GCC); it never 178 enters any package or rootfs. 179 180 Our v2 artifacts verify and install cleanly under both v2.14 (host) and 181 v3.0.8, untrusted and fully trusted (signed index + seeded key). v3 needs 182 `--usermode` for non-root `--root` installs. This becomes a real package 183 template once the toolchain work lands; until then the pin + hash above are 184 the record. 185 186 **This is no longer prose to follow by hand** -- `mk/bootstrap-host.sh` is the 187 executable form of the above. `--check` verifies presence *and version* (a 188 stale binary from an older pin is worse than a missing one, because everything 189 downstream still appears to work), and `--force` rebuilds from the pinned 190 tarballs. muon's `meson-docs`/`meson-tests` subprojects are disabled because 191 its `setup` step otherwise git-clones them from GitHub mid-build, which has no 192 place in a bootstrap driven by checksummed tarballs.