zfsbootmenu/testing/TPSReport.md

# TPS Report: Test Procedure Specification for ZFSBootMenu Release

The following list of quality checks for ZFSBootMenu should be performed on at
least two separate test environments. If possible, checks should be performed
against all supported testing environments. An initial environment can be
prepared by invoking, from `testing/`, a command like

    ./setup.sh -a -e -k -o <distro>

The `-e` flag is optional but will produce an encrypted pool with passphrase
"zfsbootmenu". By default, the `org.zfsbootmenu:keysource` property will be set
to the first boot environment configured on a pool image, which allows
ZFSBootMenu to cache keys for pools it unlocks. Testing with an encrypted pool
is recommended; all code paths encountered by an unencrypted pool should also
be hit with encrypted pools, but an encrypted pool will trigger additional code
paths that should be exercised.

The `-k` flag is also optional. It will populate the `testing/keys` directory
with your SSH `authorized_keys` file and create host keys that will be
installed in every BE to simplify SSH logins. After the `testing/keys`
directory is created, `-k` need not be specified again.

During setup, the ZFSBootMenu master branch will be cloned into `/zfsbootmenu`
within the boot environment; this environment will be used to generate a kernel
and initramfs pair in `/zfsbootmenu/build` that will be copied into the test
directory for use in the testing procedure. The newly generated images will
overwrite any existing ZBM kernel and initramfs.

Additional testbeds can be set up independently by passing the `-D <directory>`
option to the above `setup.sh` command. Alternatively, a distinct environment
can be installed on a separate pool within the same testbed by using a command
like

    ./setup.sh -i -e -p <pool-name> -o <distro>

where `<pool-name>` is a name distinct from any other pool currently present in
the testbed (the default pool name is `ztest`, but numeric components may be
appended [01, 02, etc.] to distinguish a pool from another
currently imported pool. Again, the `-e` flag is optional.

It is also possible to add a distinct boot environment to an existing pool,
using a command like

    ./setup.sh -i -x -p <pool-name> -o <distro>

where the `-x` indicates that an existing pool will be used. The option
`-p <pool-name>` is not needed if the default `ztest` pool is desired and
present. At most one BE per supported distro may be installed on a pool,
because each BE is placed at

    <pool-name>/ROOT/<distro>

For testing, a mix of multiple pools and multiple BEs on a single pool are
recommended.

Once an environment is prepared, it can be run with

    ./run -i

The `-D <directory>` argument may be provided to launch an environment in a
non-standard directory. The `-i` option is optional; it causes the directory
`${HOME}/.ssh/zfsbootmenu.d` to be created with mode 700. If this directory
exists (or is created with `-i`), `run.sh` will create a host configuration at
the path

    ${HOME}/.ssh/zfsbootmenu.d/${TESTHOST}

where `${TESTHOST}` is the basename of the test environment directory, with the
text `test.` prepended if the path does not already begin with `test.`. The
configuration enables SSH logins into the test environment simply by typing

    ssh ${TESTHOST}

The configuration will log into the root account of the testbed and bypasses
any host key verification or storage. When the test environment terminates, the
configuration file will be removed. To make use of these configurations, be
sure to add

    Include zfsbootmenu.d/*

to your `${HOME}/.ssh/config`. Once the `${HOME}/.ssh/zfsbootmenu.d` path
exists, it is not necessary to specify the `-i` option to continue to use
automatic host configuration.

ZFSBootMenu is configured by default to force the menu to appear. At least
once, test the ZFSBootMenu countdown timer and automatic boot procedure with

    ./run -A zbm.timeout=10

This should cause a 10-second countdown to appear before the default boot
environment is booted. The default boot environment is the last environment
installed in a test environment.

## Main Menu Checks

From the main menu, perform the following checks:

- [ ] Pressing `[RETURN]` on an arbitrary BE causes that environment to boot.

- [ ] Pressing `[ESCAPE]` causes the screen to redraw.

- [ ] Pressing `[CTRL+P]` displays status for the pool holding the currently
  selected boot environment.

- [ ] Pressing `[CTRL+D]` on an arbitrary BE causes its pool to be re-imported
  read/write, changing the display header to red and moving that BE to the top
  of the menu; after rebooting, this BE should be selected by default and
  should automatically boot when the timer is activated.

- [ ] Pressing `[CTRL+S]` displays the list of snapshots on the currently
  selected BE.

- [ ] Pressing `[CTRL+K]` displays the list of bootable kernels installed in the
  currently selected BE.

- [ ] Pressing `[CTRL+E]` displays the KCL edit screen, pre-populating the
  entry field with the existing KCL; changing the entry should alter the KCL
  dsplayed in the header and these changes should appear in `/proc/cmdline` in
  the next booted BE.

- [ ] Pressing `[CTRL+J]` jumps into an interactive chroot for the selected BE,
  which will be read-only if the pool is mounted read-only and read/write
  otherwise. If the pool is read-only, exiting the chroot should cause a yellow
  `[!]` warning icon to appear in the header.

- [ ] Pressing `[CTRL+R]` exits ZFSBootMenu and drops to a recovery shell.
  Exiting the recovery shell restarts ZFSBootMenu.

- [ ] Pressing `[CTRL+W]` toggles the pool holding the selected BE between
  read-only and read/write.

- [ ] Pressing `[CTRL+O]` cycles through name/creation/used sort order.

## Pool Status Checks

- [ ] Pressing `[CTRL+R]` causes a checkpoint rewind; this should be a no-op if
  the pool has not been checkpointed. Otherwise, any snapshots created after a
  checkpoint should disappear after the rewind.

- [ ] Pressing `[ESCAPE]` returns to the main menu.

## Snapshot List Checks

- [ ] Pressing `[CTRL+J]` jumps into an interactive chroot for the selected
  snapshot, or fails if the snapshot is missing a shell.

- [ ] Pressing `[CTRL+D]` shows the diff viewer:
    - Selecting one snapshot will show a diff between the snapshot and the filesystem
    - Selecting two snapshots will show the diff between them
    - The diff screen updates dynamically and is interruptable
    - After exiting the diff screen, the same snapshots will be selected

- [ ] Pressing `[RETURN]` presents the snapshot duplication interface:
    - The new name is prepopulated with `_NEW` appended to the old name
    - An existing environment with the same name triggers additional user input
    - Blanking the name and pressing `[RETURN]` aborts the duplication
    - Entering a non-empty name triggers a duplicate
    - The duplicate BE appears in the list of bootable environments
    - Neither the original nor the duplicate show a new ORIGIN property

- [ ] Pressing `[CTRL+X]` behaves as with `[RETURN]`, except:
    - Cloning does not trigger a buffered send-receive and is faster
    - The ORIGIN property of the original BE will list the selected snapshot on
      the newly cloned BE

- [ ] Pressing `[CTRL+C]` behaves as with `[RETURN]`, except:
    - Cloning does not trigger a buffered send-receive and is faster
    - The ORIGIN property of the cloned BE will list the selected snapshot on
      the original BE

- [ ] Pressing `[CTRL+N]` presents the snapshot creation interface:
    - The new snapshot name is prepopulated with %Y-%m-%d-%H%M%S
    - An existing snapshot with the same name triggers additional user input
    - Blanking the name and pressing `[RETURN]` aborts the snapshot
    - Entering a non-empty name triggers a new snapshot
    - The new snapshot appears in the list of snapshots

- [ ] Pressing `[CTRl+R]` presents the rollback creation prompt:
    - The selected snapshot is listed, highlighted in red
    - Entering anything but `ROLLBACK` cancels the operation
    - Entering `ROLLBACK` rolls the dataset back to the snapshot
    - The selected snapshot is no longer listed on the Snapshot screen

- [ ] Pressing `[CTRL+O]` cycles through name/creation/used sort order.

- [ ] Pressing `[ESCAPE]` returns to the main menu.

## Kernel List Checks

At least one BE should include multiple kernels to confirm proper functionality
of the kernel list. That BE should be selected when entering the list.

- [ ] Pressing `[RETURN]` will boot the selected kernel within the selected BE.

- [ ] Pressing `[CTRL+D]` changes the default kernel, setting the property
  `org.zfsbootmenu:kernel` on the BE to the selected kernel name, returning to
  the main menu with the selected kernel listed in the header. A subsequent
  reboot should use the selected kernel without further user interaction.

- [ ] Pressing `[CTRL+U]` clears the default kernel by inheriting the
  `org.zfsbootmenu:kernel` parameter from the BE parent.

- [ ] Pressing `[ESCAPE]` returns to the main menu.

## Common Features

- [ ] Pressing `[CTRL+L]` displays the log viewer, which should be blank unless
  a warning or error `[!]` indicator has previously appeared in the header.

- [ ] Pressing `[CTRL+H]` displays the help browser, preselecting the section
  corresponding to the current menu screen.

- [ ] Pressing the left/right arrow keys moves to the next/previous screen shown
  in the header. Screens will not wrap around.

- [ ] When booting a test VM with GTK output and efifb, the font will be resized
  to change the apparent resolution of the console. Successful font changes are
  logged via `zinfo()`.

## General Testing

- [ ] If every encryption root specifies an `org.zfsbootmenu:keysource`
  property and a `file://` key location, toggling pools between read-only and
  read/write should proceed without requesting passwords.

- [ ] If any encryption root specifices an `org.zfsbootmenu:keysource`
  property and a `file` key location, keys should appear under the path

      /zfsbootmenu/.keys/<keysource>/<keylocation>

  where `<keysource>` is the value of an `org.zfsbootmenu:keysource` property
  on an encryption root and and `<keylocation>` is its key location.

- [ ] If any encryption root lacks an `org.zfsbootmenu:keysource` property,
  toggling its pool between read-only and read/write should force a passphrase
  prompt each time. (Keys must not have been already cached before a keysource
  property is cleared, or the previously cached keys will continue to be used.)

- [ ] ZBM can be forced to spin until the pool defined via `zbm.prefer=pool!`
  is available. During the wait period, the user can exit to a recovery shell
  via `[ESCAPE]`.

- [ ] ZBM can be forced to import the pool defined via `zbm.prefer=pool!!` with
  all other pools ignored/skipped. This will spin until the pool is available, and
  can be broken via `[ESCAPE]`.

- [ ] Invalid `spl_hostid` or `spl.spl_hostid` values do not cause ZBM to fail.

- [ ] Adding `zbm.skip` to the KCL causes ZBM to immediately boot BOOTFS,
  otherwise show the main menu.

- [ ] Adding `zbm.show` to the KCL causes ZBM to always show the main menu.

- [ ] Adding `rd.vconsole.keymap=fr` causes the ZBM keymap to change when booted
  in GTK mode. This can be verified by dropping to the recovery shell and typing
  qwerty.

- [ ] User hooks are installed from `$zfsbootmenu_hook_root` as defined in the
  dracut or mkinitcpio configuration used by `generate-zbm`.

- [ ] Any hooks installed in a ZFSBootMenu image execute at expected points.

- [ ] Installed hooks cam be properly skipped by specifying their names in the
  `zbm.skip_hooks` kernel command-line parameter.

## Recovery Shell and SSH Access

- [ ] Basic tab completion for some internal functions is available in the
  recovery shell.

- [ ] SSH'ing in to dracut-crypt enabled ZBM build should result in a functional
  shell, with a proper path and prompt set.

- [ ] A running copy of ZBM can be taken over from a remote session:

    - Executing `zbm` or `zfsbootmenu` from the remote login will attempt to
      stop any currently running instance.

    - Until the running instance is stopped, the new invocation will spin with
      a timed message until it succeeds or the user cancels the attempt.

- [ ] In the recovery shell, in a recovery image, verify networking:
    - `dhclient` should successfully configure a network interface.
    - `curl` should successfully retrieve https://get.zfsboot.menu/efi
    - `mbuffer -I 1234` should start and listen on port 1234, accepting traffic

## OS-Specific Image Creation

- [ ] For each supported distribution [Void, Void Musl, Arch, Ubuntu, Debian],
  verify that `module-setup.sh` is able to correctly install all required
  binaries in the initramfs.

- [ ] For each supported distribution, verify that `generate-zbm` can successfully
  produce an versioned and unversioned initramfs and a unified EFI bundle.

- [ ] For each supported distribution, verify that the components and EFI bundles
  are able to correctly boot other systems. The check stages listed above should
  be used and any functionality that is missing or broken noted.