ANITA(1)

NAME
anita - Automated NetBSD Installation and Test Application

SYNOPSIS
anita [--workdir work_directory] [--vmm qemu | xl | xm | gxemul | simh] [--vmm-args vmm_arguments] [--disk-size size] [--run command] [--sets sets] [--test-timeout timeout] [--persist] [--boot-from cdrom | floppy] [--structured-log] [--structured-log-file file] [--tests kyua | atf] [--dtb dtb] [--image-format format] [--machine machine] [--partitioning-scheme scheme] [--xen-type pv | pvshim | hvm | pvh] [--no-entropy] mode URL

DESCRIPTION
anita is a tool for automated testing of the NetBSD installation procedure and of NetBSD in general. Using anita, you can fully automate the process of downloading a NetBSD distribution, installing it in a virtual machine, booting the installed system, and running the NetBSD test suite.

The NetBSD ports currently supported as targets (i.e., as the system to install and run under emulation) are i386, amd64, sparc, sparc64, evbarm-earmv7hf, evbarm-aarch64, pmax, hpcmips, vax, hppa, macppc, alpha, and riscv-riscv64. The host (the system running anita) can be any NetBSD port or even a different Unix-like system such as Linux, FreeBSD, or macOS.

i386 and amd64 targets can be run under either qemu or Xen. Other targets use either qemu, gxemul, or simh depending on the target architecture.

EXAMPLES
To install NetBSD 10.0/i386 from the master NetBSD FTP site, enter

    anita install http://ftp.netbsd.org/pub/NetBSD/NetBSD-10.0/i386/

Installing NetBSD/sparc, sparc64, vax, or hppa works a bit differently: they use an ISO image instead of a directory containing boot floppies and sets:

    anita install http://ftp.netbsd.org/pub/NetBSD/iso/10.0/NetBSD-10.0-sparc.iso

The evbarm and riscv ports do not currently support installation using sysinst, so anita will instead install them by decompressing the pre-built disk image from the release onto the beginning of the virtual disk. The target will then be rebooted to trigger the automatic image resizing that happens on first boot, and after that, these ports work the same as the other ports. To download an evbarm or riscv image, use a command like the following:

    anita install http://nycdn.netbsd.org/pub/NetBSD-daily/HEAD/latest/evbarm-aarch64/
    anita install http://nycdn.netbsd.org/pub/NetBSD-daily/HEAD/latest/riscv-riscv64/

To boot the newly installed hard disk image and get a login prompt, replace install with interact:

    anita interact http://ftp.netbsd.org/pub/NetBSD/NetBSD-10.0/i386/

When you are done interacting with the virtual machine, you can kill it by typing control-a x (qemu) or control-c (gxemul).

To run the NetBSD test suite on the installed system, use

    anita test http://ftp.netbsd.org/pub/NetBSD/NetBSD-10.0/i386/

If you have a recent NetBSD host and a qemu version that supports NVMM (the NetBSD Virtual Machine Monitor), you can use it to run the tests several times faster than using qemu's built-in emulation:

    anita --vmm-args="-accel nvmm" test http://ftp.netbsd.org/pub/NetBSD/NetBSD-10.0/i386/

To install a snapshot, use something like the following (adjusting the version number in the sparc URL as needed):

    anita install http://nycdn.netbsd.org/pub/NetBSD-daily/HEAD/latest/i386/
    anita install http://nyftp.netbsd.org/pub/NetBSD-daily/HEAD/latest/images/NetBSD-10.99.7-sparc.iso

If you have built a NetBSD release locally using build.sh -R, you can point anita directly at the RELEASEDIR or ISO using a "file:" URL:

    anita install file:///path/to/releasedir/i386/

or simply use an absolute pathname as shorthand for the above:

    anita install /path/to/releasedir/i386/

HOW IT WORKS
anita works by "screen scraping" the sysinst output over an emulated serial console, recognizing sysinst prompts and sending canned responses.

anita will create a work directory and cache the distribution sets, boot floppies, and a hard disk image in it. By default, the work directory is a subdirectory of the current working directory, with a unique name automatically generated from the distribution URL, for example,

    work-http---ftp.netbsd.org-pub-NetBSD-NetBSD-10.0-i386-+a4c39

You can also specify the name of the work directory explicitly using the --workdir option.

When you rerun anita with the same URL, files that already exist in the work directory will not be re-downloaded or rebuilt, so if you run you anita install with the same URL twice in a row, the second run will effectively be a no-op, and if you rerun anita interact, the system will be booted from the existing disk image, skipping the installation stage. To force things to be redone, simply remove the work directory. If you remove just the hard disk image file wd0.img, anita install will recreate it from the cached distribution files.

To ensure that the cached system state is always that of a freshly installed system, anita enables the qemu snapshotting feature for the system disk by default (but not for other disks). Therefore, if you log in and make changes to the system, they will not be saved to the disk image file. To override this behavior, specify the --persist option.

MODES
The operation performed by anita is determined by the mode argument, which takes the following values:

install
    Install NetBSD if not already installed.

boot
    Install NetBSD if not already installed, then boot the installed system to the login prompt.

interact
    Install NetBSD if not already installed, boot it, and then connect the serial console to the terminal for interactive use. The qemu escape character control-a is in effect; for example, you can use control-a x to exit, control-a c to enter the qemu monitor, or control-a b to send a break (useful for entering DDB). Interacting with the boot blocks and other aspects of the boot process is possible unless the --run option has been specified. With --run, the interactive session only starts once the --run command has been successfully run.

test
    Install NetBSD if not already installed, then boot it and run the test suite from /usr/tests. By default, the test suite is run using ATF. If NetBSD was built with ‘MKKYUA=yes’, you can use Kyua instead by passing the option --tests kyua. The raw output of the test execution and various other test reports are stored in a tests/ subdirectory under the work directory.

    When Kyua is used to run the test suite, the tests/ subdirectory contains the raw output of the tests execution, a copy of the Kyua database in the store.db file with the results of the tests, and an itemized HTML report in the html/ subdirectory.

    When the ATF tools are used to run the test suite, the atf-report output will be displayed on standard output and the following output files are placed in the tests/ subdirectory: the raw atf-run output in test.tps, the output from atf-report -o xml in test.xml, and the output from atf-report -o ticker in test.txt. To facilitate the further processing of the XML output into HTML, the files tests-results.xsl, tests-results.dtd, and tests-results.css are also included.

print-workdir
    Print the pathname of the work directory on standard output. This is intended for use by scripts that need to access files in the work directory, particularly when the --workdir option is not used but the name of the directory is automatically generated.

OPTIONS
The following command line options are supported:

--workdir directory
    The work directory. The default is an automatically generated name under ".".

--vmm qemu | xl | xm | gxemul | simh
    Specify the virtual machine monitor. The default virtual machine monitor is qemu. If the target system architecture is not supported by qemu, anita will automatically switch to gxemul or simh as needed. If anita is running in a Xen dom0 and the target system architecture is i386 or amd64, it is also possible to specify xl or xm, which will make anita install the target system in a Xen domU using the current xl or the historic xm interface, respectively. This requires running anita as root. Any changes made to the system disk image will be persistent whether or not the --persist option was given. The Xen support should be considered experimental and may not be fully tested. For backwards compatibility, xen is accepted as a synonym for xm.

--vmm-args string
    Additional arguments to pass to the virtual machine monitor (e.g., qemu). The arguments are given as a single string, which may contain multiple arguments separated by whitespace. There is no way to pass an argument containing whitespace. This option was formerly called --qemu-args; the old name is still accepted for backwards compatibility.

--disk-size size
    The size of the virtual disk NetBSD gets installed on. The default is large enough to hold the OS installation itself when also using default values for the --sets and --memory-size options, but if you need additional space, you can specify a larger size. The size is given in bytes, or a suffix of k, M, G, or T can be used for kilo-, mega-, giga-, or terabytes.

--memory-size size
    The size of the virtual RAM. The size is given in bytes, or a suffix of k, M, or G can be used as with the --disk-size option. The default is architecture dependent. Note that since sysinst sizes the swap partition based on the amount of RAM, if you run anita install with a large --memory-size, you may also have to increase --disk-size.

--run command
    Log in to the virtual machine as root and execute the given shell command in it once it has booted. This is only meaningful when used with the boot or interact command. Since the command is sent to an interactive shell over the console tty, it should be kept short and simple to avoid running into tty limitations or quoting issues. Complex commands may be executed by preparing a disk image containing a file system containing a shell script, and specifying something like

        --vmm-args '-hdb disk.img' --run 'mount /dev/wd1a /mnt && /mnt/script'

    The shell command is run using /bin/sh regardless of the login shell of the root user on the target system. The exit status of the shell command is returned as the exit status of anita.

--sets sets
    The distribution sets to install, as a comma-separated list. For a minimal install, use something like

        --sets kern-GENERIC,modules,base,etc

    A kernel, base, and etc must always be included.

--run-timeout timeout
    Set a timeout for the tests run using the --run option, in seconds. The default is 3600 seconds (one hour).

--test-timeout timeout
    Set a timeout for the tests run in the test mode, in seconds. Starting with anita version 2.2, the timer is reset at the beginning of each ATF test program, so the timeout only needs to be greater than the duration of the longest test program rather than the full test run. The default is 10800 seconds (3 hours).

--persist
    Store any changes to the contents of the system disk persistently, such that they may affect future anita runs, instead of the default behavior where only the install mode can modify the disk contents and all other modes work with an ephemeral snapshot copy of the freshly installed system.

--boot-from cdrom | floppy | kernel
    For architectures that support booting from more than one type of media (typically CD-ROM or floppies), specify which one to use. Most architectures only support one type of boot media, and specifying an unsupported typ will cause the boot to fail. Due to limitations of the emulators or their emulated firmware, some architectures cannot be booted from install media at all, and instead boot by passing a kernel directly to the emulator. The default is floppy for i386, kernel for alpha, and cdrom for all other targets.

--structured-log
    Generate log output in a structured format similar to Python code. Commands sent to the serial console by anita are logged as send(t, 'command...'), where t is a timestamp in seconds since the Unix epoch. Data received are logged as recv(t, 'data...'). Calls to pexpect's expect() function are logged as expect(t, 'regexp...'), and the actual strings matched by them as match(t, '...'). Unprintable characters in the data strings are escaped using Python string syntax.

    The default is to do unstructured logging where the raw output from the virtual machine console is sent to standard output as-is, and the commands sent to the console are only logged if echoed by the virtual machine.

--structured-log-file
    Like --structured-log, but logs to a given file rather than to standard output, and in addition to rather than instead of the default unstructured logging.

--tests kyua | atf
    The test framework to use for running tests. The default is atf.

--dtb
    The location of the Device Tree Blob file, needed with the evbarm-earmv7hf port when using the default vexpress-a15 machine type. The default is the location of the file vexpress-v2p-ca15-tc1.dtb in the dtb-arm-vexpress package assuming both it and anita itself have been installed via pkgsrc.

--xen-type type
    Select the type of virtualization to use with Xen. This can be pv, pvshim, hvm, or pvh. The default is pv.

--image-format format
    The disk image format to use for the virtual machine's system disk. Supported values are dense, a raw disk image that has been fully preallocated by writing zeros, and sparse, a raw disk image with holes (when supported by the underlying file system). The default is dense.

--machine machine
    The machine type to emulate. This may be used with the evbarm-earmv7hf port to select the virt qemu machine type instead of the default of vexpress-a15. Not consistently supported for other ports and VMMs.

--partitioning-scheme scheme
    Use the given partitioning scheme instead of the port's default one. Currently only supported for the i386 and amd64 ports, where a scheme of MBR can be selected instead of the default of GPT.

--no-entropy
    If sysinst prompts the user to enter entropy and offers an option not to enter it, select that option. This is intended for testing the behavior of said option and the resulting installation, and only works with certain versions of NetBSD that offer such an option. The default is to supply the guest being installed with entropy from the host.

DEBUGGING NETBSD USING ANITA
anita can serve as a convenient platform of installing and booting NetBSD for debugging purposes. When doing this, it is useful to build NetBSD with debug symbols and to install the source on the virtual machine to enable source-level debugging.

For more details, please see
    http://wiki.netbsd.org/kernel_debugging_with_qemu/.

SEE ALSO
atf-report(1), atf-run(1), qemu(1), kyua(1), tests(7)

BUGS IN ANITA
anita supports only a limited number of NetBSD ports. There may be other ports with working emulators, and support should be added for those. Patches are welcome.

anita is likely to break whenever any significant change is made to the sysinst user interface.

Installing NetBSD releases older than 2.1 has not been tested.

Exporting ATF test reports to the host system does not yet work with the evbarm-earmv7hf or hpcmips ports for lack of support for a second block device.

BUGS IN NETBSD

NetBSD/i386 releases older than 4.0 will install, but when booting the installed image, they hang after the "root on ffs" message.

NetBSD/i386 versions older than 2009-06-13 13:35:11 fail to find any PCI buses when run under qemu; see PRs 38729 and 42681.

NetBSD/vax is unable to run the ATF tests as of source date 2018.03.22.12.16.11.

Timing is off by a factor of two when running on NetBSD hosts; see PR 43997.

For current reports of other NetBSD bugs found using anita, see

    http://releng.netbsd.org/test-results.html

BUGS IN QEMU

Versions of qemu known to work with anita on NetBSD hosts are 0.15, 1.2, 1.4.1, and 2.0.0nb4 or newer.

Some floating point tests fail under qemu but not on real hardware, presuambly due to bugs qemu's floating point emulation. In particular, floating point exceptions are broken when using the default TCG acceleration, as reported in

    https://gitlab.com/qemu-project/qemu/-/issues/215

Running multithreaded programs (such as the NetBSD test suite) on an emulated i386 or amd64 system used to require qemu patches that were in pkgsrc beginning with qemu 0.12.3nb3. They were finally integrated into qemu on 2011-12-11. See PR 42158 and

    https://bugs.launchpad.net/bugs/569760

for details.

When attempting to install NetBSD-current in qemu 1.0, it panics during the install kernel boot due to a regression in qemu's emulation of the PCI configuration registers. The work-around is to use qemu 0.xx. See PR 45671 and https://bugs.launchpad.net/qemu/+bug/897771 for details. This bug has since been fixed on the qemu mainline.

In addition to the above, there have been several further regression in the 1.x series of qemu that have impacted anita:

    https://bugs.launchpad.net/qemu/+bug/1089996
    https://bugs.launchpad.net/qemu/+bug/1091241
    https://bugs.launchpad.net/qemu/+bug/1127369
    https://bugs.launchpad.net/qemu/+bug/1154328

These are believed to be fixed in qemu 1.5.

Installing NetBSD 5 or older on i386 or amd64 takes a long time with recent versions of qemu because the bootloader countdown runs at 1/20 the normal speed, and there is a long delay between loading the kernel and the kernel printing its first console output, which can easily be mistaken for a hang. Please be patient. This issue has been worked around in NetBSD 6 and newer; see PR 43156 for details.

Installing using qemu version 1.5.1, 1.6.0, 1.7.0, or 2.0.0 prior to 2.0.0nb4 on NetBSD fails due to the serial console dropping characters; see PR 48071 and the qemu bug reports

    https://bugs.launchpad.net/qemu/+bug/1335444
    https://bugs.launchpad.net/qemu/+bug/1399943

The same qemu versions work when hosted on Linux, but only by accident. This bug is fixed for i386 and amd64 targets in qemu 2.0.0nb4 in pkgsrc and the qemu 2.1 release, and finally fixed for sparc in qemu 2.8.

Sending a break sequence to the serial console using "control-a b" was broken, fixed, broken again, fixed again, broken again as the fix was reverted for causing another regression, and finally fixed again in qemu 3.0:

    https://bugs.launchpad.net/qemu/+bug/1654137

Interacting with the boot blocks over the serial console is also currently broken:

    https://bugs.launchpad.net/qemu/+bug/1743191

The qemu -icount option looks useful for making the tests less dependent on host timing, and its sleep=on|off argument might be used to speed up the tests by emulating the passage of time instead of waiting for actual time to pass. This is now partly functional as

    https://bugs.launchpad.net/qemu/+bug/1774677

has been fixed, but causes spurious IDE disk errors in the guest. Qemu also advertises record/replay capability, but it does not actually work, as reported in

    https://bugs.launchpad.net/qemu/+bug/1810590

In qemu 5.1.0, booting NetBSD/sparc did not work:

    https://bugs.launchpad.net/qemu/+bug/1892540

This is fixed since 5.2.0.

BUGS IN KVM

When anita is run on a Linux host using qemu-kvm 0.12.3, and is used to boot a version of NetBSD-current newer than 2009-11-04 14:39:17, the emulated NetBSD system hangs during boot; see PR 44069 for details. This issue can be worked around by passing Anita the command line option --vmm-args -no-kvm to disable kvm. The alternative --vmm-args -no-kvm-irqchip performs better but doesn't quite work: the system installs and boots, but the test suite occasionally fails to complete; see PR 44176.

As of 2020, the above issue has been fixed.

BUGS IN PYTHON

Versions of anita prior to 1.40 may get the error

    [Errno ftp error] 200 Type set to I

during the downloading of distribution sets; this is a regression in Python 2.7.12:

    http://bugs.python.org/issue27973

This problem is worked around in anita 1.40.