Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
32 changes: 29 additions & 3 deletions COVERAGE.md
Original file line number Diff line number Diff line change
Expand Up @@ -11,15 +11,15 @@ acceptance test of an appliance recipe, docs/org-plan.md section 1).
| overlay/usr/lib/inithooks/lib/nodebb.sh | tests/nodebb.bats (25 tests) | 100 percent (43/43) under kcov | every function and every branch |
| overlay/usr/lib/inithooks/firstboot.d/40nodebb | tests/hook.bats (15 tests) | 96.97 percent (32/33) under kcov | the one uncovered line is inside the dialog loop, which needs a terminal |
| overlay/etc/nginx/* | tests/nginx.bats (12 tests) | not executable | asserted as content: the geo variable, the map, the listeners, the proxy headers |
| tests/lib/boot-test-lib.sh | tests/boot-test.bats (34 tests) | 100 percent (124/124) under kcov | the logic of the boot test: argument parsing, address discovery, deadlines, the HTTP and diff verdicts |
| tests/lib/boot-test-lib.sh | tests/boot-test.bats (41 tests) | 100 percent (137/137) under kcov | the logic of the boot test: argument parsing, address discovery, deadlines, the HTTP and diff verdicts |
| bin/keel-archive-check | tests/archive-check.bats (8 tests) | 100 percent (26/26) under kcov | the build time check that the archive copy in the build tree is the live archive |
| conf.d/zz-project-packages | tests/project-packages.bats (13 tests) | 100 percent (29/29) under kcov | the build time check that each project package is the candidate of the archive, and a project build |
| overlay/usr/lib/inithooks/bin/nodebb.py | none | 0 | dialog wrapper, only reached with a terminal attached |
| conf.d/main | tests/boot-test.sh (build step) | integration only | build time script, 0004 pragmatic limits |
| tests/boot-test.sh | itself | integration only | the thin main of the acceptance test: keel and LXC as root |

Total over the five measured shell files: 99.22 percent (254/256) before the
terminal test, 99.61 percent (255/256) with it.
Total over the five measured shell files: 99.63 percent (267/268) before the
terminal test, 100 percent (268/268) with it.

`tests/coverage.sh` runs the whole bats suite under kcov, measures the
library, the first boot hook, the boot test's own library and the two build
Expand All @@ -34,6 +34,32 @@ LXC and runs `tests/boot-test.sh` against it. Nothing is built there: the
runner has no fab, deck or buildtasks. That job produces the check
`appliance / build-and-boot`.

### What the gate found once the layer booted (2026-09-27)

With the republished chain the gate pulls, verifies, assembles and boots, and
the whole hook chain runs. Three defects surfaced behind the one that had been
hiding them, and all three are fixed:

1. `redis-server.service` failed with `status=226/NAMESPACE`: the container
config asked for no apparmor profile, so systemd could not give the unit a
mount namespace.
2. nginx refused to start on `ssl_ciphers 'ZZ_SSL_CIPHERS'`, an unsubstituted
mark. Fixed in buildtasks (`layer_needs_ssl_ciphers` now scans the child's
overlays) and the layer was rebuilt.
3. `firstboot.d/40nodebb` never returned. The layer ships the plain appliance
`inithooks.service`, which runs the hooks with `StandardOutput=tty` on
`/dev/tty1`; a container image gets a unit that logs to syslog and the
console instead. Nothing reads tty1 in a container nobody has attached to,
so `./nodebb setup` filled the terminal buffer and blocked in
`n_tty_write`. `bt_mark_container` now does what a container build does:
the marker, `REDIRECT_OUTPUT=true`, and a drop-in that puts the unit's
output on the journal.

Measured on the build host against the published chain, with all three in
place: the first boot completes through `98finalize`, `[40nodebb]
successfully completed`, port 80 answers 307 to https and port 443 answers
200 with `<title>Home | NodeBB</title>` over IPv6.

State of that check on 2026-09-26: **failing on a defect of the published
layer, so it is not a required status yet.** The `nodebb` layer on
the mirror (sha256 `40e3f4da`) carries the build time wrappers
Expand Down
13 changes: 10 additions & 3 deletions tests/README.md
Original file line number Diff line number Diff line change
Expand Up @@ -64,9 +64,16 @@ What it does, in order:

1. `keel pull` and `keel assemble` the chain (core, nodejs-nginx, nodebb)
into `<lxc-path>/<name>/rootfs`.
2. Creates `var/lib/turnkey-info/inithooks.service/lxc` in the rootfs, the
marker `bt-container` writes and the one `keel inspect` reads to call the
machine a container (`network.managed_by: host`).
2. Marks the tree as a container build, which is what
`bt_mark_container` does and what buildtasks' `patches/container/conf`
does for a real container image: the marker
`var/lib/turnkey-info/inithooks.service/lxc` that `keel inspect` reads to
call the machine a container (`network.managed_by: host`),
`REDIRECT_OUTPUT=true` in `etc/default/inithooks`, and a drop-in that
gives `inithooks.service` `StandardOutput=journal`. Without the last two
the hooks write to `/dev/tty1`, which nobody reads in a container, and the
first hook that prints more than the terminal buffer holds blocks there
forever.
3. Writes a random `root_password` and `app_password` under
`etc/keel/secrets` (mode 0600) and installs `tests/instance.yaml` at
`etc/keel/instance.yaml` and `etc/inithooks.yaml`.
Expand Down
66 changes: 66 additions & 0 deletions tests/boot-test.bats
Original file line number Diff line number Diff line change
Expand Up @@ -247,6 +247,72 @@ never() { return 1; }
[[ $output == *"lxc.apparmor.allow_nesting = 1"* ]]
}

fake_rootfs() {
# fake_rootfs [VALUE]: a scratch rootfs with an inithooks defaults file,
# REDIRECT_OUTPUT set to VALUE (default false), printed on stdout
local rootfs="$BATS_TEST_TMPDIR/rootfs-$RANDOM"
mkdir -p "$rootfs/etc/default"
cat > "$rootfs/$BT_INITHOOKS_DEFAULT" <<DEF
INITHOOKS_CONF=/etc/inithooks.conf
RUN_FIRSTBOOT=true
REDIRECT_OUTPUT=${1-false}
SUDOADMIN=false
DEF
printf '%s\n' "$rootfs"
}

@test "mark_container: writes the marker the unit conditions and inspect read" {
rootfs=$(fake_rootfs)
run bt_mark_container "$rootfs"
[ "$status" -eq 0 ]
[ -f "$rootfs/var/lib/turnkey-info/inithooks.service/lxc" ]
}

@test "mark_container: turns REDIRECT_OUTPUT on, so no hook blocks writing to tty1" {
rootfs=$(fake_rootfs false)
run bt_mark_container "$rootfs"
[ "$status" -eq 0 ]
grep -q '^REDIRECT_OUTPUT=true$' "$rootfs/$BT_INITHOOKS_DEFAULT"
# the rest of the file is left alone
grep -q '^RUN_FIRSTBOOT=true$' "$rootfs/$BT_INITHOOKS_DEFAULT"
grep -q '^SUDOADMIN=false$' "$rootfs/$BT_INITHOOKS_DEFAULT"
}

@test "mark_container: takes the first boot off tty1 with a systemd drop-in" {
rootfs=$(fake_rootfs)
run bt_mark_container "$rootfs"
[ "$status" -eq 0 ]
dropin="$rootfs/$BT_INITHOOKS_DROPIN"
[ -f "$dropin" ]
grep -q '^\[Service\]$' "$dropin"
grep -q '^StandardOutput=journal$' "$dropin"
grep -q '^StandardError=journal$' "$dropin"
}

@test "mark_container: a tree that already redirects is left redirecting" {
rootfs=$(fake_rootfs true)
run bt_mark_container "$rootfs"
[ "$status" -eq 0 ]
[ "$(grep -c '^REDIRECT_OUTPUT=true$' "$rootfs/$BT_INITHOOKS_DEFAULT")" -eq 1 ]
}

@test "mark_container: a rootfs with no inithooks defaults fails loudly" {
rootfs="$BATS_TEST_TMPDIR/bare"
mkdir -p "$rootfs"
run bt_mark_container "$rootfs"
[ "$status" -eq 1 ]
[[ "$output" == *"is not in the rootfs"* ]]
}

@test "mark_container: defaults that declare no REDIRECT_OUTPUT fail loudly" {
rootfs=$(fake_rootfs)
grep -v REDIRECT_OUTPUT "$rootfs/$BT_INITHOOKS_DEFAULT" > "$rootfs/trimmed"
mv "$rootfs/trimmed" "$rootfs/$BT_INITHOOKS_DEFAULT"
run bt_mark_container "$rootfs"
[ "$status" -eq 1 ]
[[ "$output" == *"declares no REDIRECT_OUTPUT"* ]]
}

@test "spec_targets: both paths the first boot reads, under the rootfs" {
output=$(bt_spec_targets /r)
[ "$output" = $'/r/etc/keel/instance.yaml\n/r/etc/inithooks.yaml' ]
Expand Down
15 changes: 8 additions & 7 deletions tests/boot-test.sh
Original file line number Diff line number Diff line change
Expand Up @@ -54,14 +54,15 @@ mkdir -p "$BT_ROOTFS"
keel pull "$BT_APPLIANCE" --source "$BT_LAYERS_DIR" --cache-dir "$BT_CACHE_DIR" --non-interactive
keel assemble "$BT_APPLIANCE" --rootfs "$BT_ROOTFS" --cache-dir "$BT_CACHE_DIR" --non-interactive

# 2. The container marker, the instance spec, the secrets it references and
# the conf the first boot hooks read. The marker under
# /var/lib/turnkey-info is what bt-container writes and what inspect reads
# to call the machine a container (managed_by: host); the conf is what
# makes the first boot headless, and without it 30rootpass and 40nodebb
# wait on a dialog forever.
# 2. The container marks, the instance spec, the secrets it references and
# the conf the first boot hooks read. bt_mark_container does what
# buildtasks' container patch does: the marker under /var/lib/turnkey-info
# that inspect reads to call the machine a container (managed_by: host),
# and REDIRECT_OUTPUT=true, without which a hook that prints a lot blocks
# writing to a tty1 nobody reads. The conf is what makes the first boot
# headless; without it 30rootpass and 40nodebb wait on a dialog forever.
log "installing the spec, the secrets and the conf into $BT_ROOTFS"
install -D -m 0644 /dev/null "$BT_ROOTFS/var/lib/turnkey-info/inithooks.service/lxc"
bt_mark_container "$BT_ROOTFS"
install -d -m 0700 "$BT_ROOTFS/etc/keel/secrets"
for target in $(bt_secret_targets "$BT_ROOTFS"); do
bt_random_password > "$target"
Expand Down
47 changes: 47 additions & 0 deletions tests/lib/boot-test-lib.sh
Original file line number Diff line number Diff line change
Expand Up @@ -26,6 +26,12 @@ BT_SECRETS="root_password app_password"
# the final name is a maintainer decision (brief section 11), so the test
# installs the same file at both until then.
BT_SPEC_PATHS="etc/keel/instance.yaml etc/inithooks.yaml"
# What marks the tree as a container build, relative to the rootfs: the
# marker file bt-container writes and the inithooks defaults whose
# REDIRECT_OUTPUT it sets. See bt_mark_container.
BT_CONTAINER_MARKER="var/lib/turnkey-info/inithooks.service/lxc"
BT_INITHOOKS_DEFAULT="etc/default/inithooks"
BT_INITHOOKS_DROPIN="etc/systemd/system/inithooks.service.d/container.conf"

bt_usage() {
cat <<USAGE
Expand Down Expand Up @@ -239,6 +245,47 @@ lxc.start.auto = 0
CONFIG
}

bt_mark_container() {
# bt_mark_container ROOTFS: make the tree look like the container build
# buildtasks produces, which is two things, both from its
# patches/container/conf:
#
# the marker under /var/lib/turnkey-info, which inithooks' unit
# conditions read and which `keel inspect` reads to call the machine a
# container (network.managed_by: host), and
#
# REDIRECT_OUTPUT=true in /etc/default/inithooks, which sends first
# boot output to the log with a tail on the active console instead of
# writing it straight to tty1.
#
# and a drop-in that keeps the first boot off tty1.
#
# The last two are not cosmetic. The layer ships the plain appliance
# inithooks.service, which runs the hooks with StandardOutput=tty on
# /dev/tty1; the unit a container image gets instead logs to syslog and
# the console. Nothing reads tty1 in a container nobody has attached to,
# so a hook that prints more than the terminal buffer holds blocks in
# the write and never returns. `./nodebb setup` prints well past that:
# it hung firstboot.d/40nodebb forever with the node process asleep in
# n_tty_write, and the gate saw a first boot that never finished.
local rootfs=$1 defaults=$1/$BT_INITHOOKS_DEFAULT
install -D -m 0644 /dev/null "$rootfs/$BT_CONTAINER_MARKER" || return 1
if [ ! -f "$defaults" ]; then
echo "boot-test: $defaults is not in the rootfs" >&2
return 1
fi
sed -i '/REDIRECT_OUTPUT/ s/=.*/=true/' "$defaults" || return 1
if ! grep -q '^REDIRECT_OUTPUT=true$' "$defaults"; then
echo "boot-test: $defaults declares no REDIRECT_OUTPUT to set" >&2
return 1
fi
install -D -m 0644 /dev/stdin "$rootfs/$BT_INITHOOKS_DROPIN" <<DROPIN || return 1
[Service]
StandardOutput=journal
StandardError=journal
DROPIN
}

bt_spec_targets() {
# bt_spec_targets ROOTFS: the paths the spec is installed at.
local relative
Expand Down
Loading