Understanding Checkbox

Checkbox by itself doesn’t test anything. It uses unit definitions grouped in providers to actually run or do something.

To better understand how Checkbox works let’s concentrate on the relationship between the following entities:

  • test command (program that is invoked as a test)

  • job unit

  • test plan

  • provider

  • launcher

Test Command

Automated and some interactive tests use external commands to help determine the outcome of the test. For instance the command:

ping 8.8.8.8 -c 1

Will check whether the device can ping a public DNS. The command returns 0 on success (ping came back), and 1 if the ping timed out.

Let’s turn this simple command into a Checkbox test.

Job Unit

The command from the above paragraph can now be used in a Job Unit:

id: ping-public-dns
_summary: Ping public DNS
plugin: shell
command: ping 8.8.8.8 -c 1

Notice how the command field is a straight copy-paste of the Test Command

RELATIONSHIP: Test Command is a part of a Job Unit

Note

Some test are fully manual and don’t run any commands.

See Checkbox Unit Types for more info on job units

Test Plan

When Checkbox is run from the command line without any parameters, i.e.:

$ checkbox-cli

It doesn’t present all the tests available in the system. Checkbox asks the user which Test Plan to use.

Note

For how to directly run hand-picked jobs see: Run a particular test plan or a set of jobs.

Test Plans are units for grouping related jobs together. They also provide a mechanism for creating new jobs in runtime. The phase in which this is done is known as bootstrapping. Good example for when the bootstrapping is needed is testing multi-GPU system. No one knows upfront which GPU(s) will be present in the system so bootstrapping phase will instantiate appropriate job units from template units.

RELATIONSHIP: Test Plan includes a job that can be run.

RELATIONSHIP: Test Plan can “generate” a job (through bootstrapping) that can be run.

You can read more about templates here: Template Unit, and about test plans here: Test Plan Unit.

Provider

In order for Checkbox to see any jobs, test plans, and other units, those units need to be written to a .pxu file located in a unit subdirectory of a provider available in the system.

RELATIONSHIP: Units are placed in a Provider

See tutorials for a tutorial on how to create a provider from scratch.

Launcher

Launchers can be used to make it easier to run Checkbox in a preset way.

Those can for instance preset:

  • which test plan to use

  • whether the session should be interactive or automated

  • which tests to exclude

  • how and where to submit the results

There is a full launcher reference here: Checkbox launchers.