Test Plan Unit

The test plan unit is a facility that describes a sequence of jobs or templates that should be executed together.

Jobs are selected by either listing their identifier or a regular expression that matches their identifier. Listing a template identifier will select all the jobs instantiated by it. Selected jobs are executed in the sequence they appear in the list, unless they need to be reordered to satisfy dependencies which always take priority. It is also possible to exclude jobs from the selection using the same principles.

Test plans can be nested, so you can define several smaller test plans and combine them into a bigger one, which helps with maintenance and flexibility.

Test plans can contain additional meta-data which can be used in a graphical user interface. You can assign a translatable name and description to each test plan.

Test plans are also typical units so they can be defined with the familiar RFC822-like syntax that is also used for job definitions. There can also be multiple test plan definitions per file, just like with all the other units, including job definitions.

Test Plan Fields

The following fields can be used in a test plan. Note that not all fields need to be used or even should be used. Please remember that Checkbox needs to maintain backwards compatibility so some of the test plans it defines may have non-typical constructs required to ensure proper behavior. You don’t have to copy such constructs when working on a new test plan from scratch

id:

Each test plan needs to have a unique identifier. This is exactly the same as with other units that have an identifier (like job definitions and categories).

This field is not used for display purposes but you may need to refer to it on command line so keeping it descriptive is useful

name:

A human-readable name of the test plan. The name should be relatively short as it may be used to display a list of test plans to the test operator.

Remember that the user or the test operator may not always be familiar with the scope of testing that you are focusing on. Also consider that multiple test providers may be always installed at the same time. The translated version of the name (and icon, see below) is the only thing that needs to allow the test operator to pick the right test plan.

Please use short and concrete names like:

  • “Storage Device Certification Tests”

  • “Ubuntu Core Application’s Clock Acceptance Tests”

  • “Default Ubuntu Hardware Certification Tests”.

The field has a soft limit of eighty characters. It cannot have multiple lines. This field should be marked as translatable by prepending the underscore character (_) in front. This field is mandatory.

description:

A human-readable description of this test plan. Here you can include as many or few details as you’d like. Some applications may offer a way of viewing this data. In general it is recommended to include a description of what is being tested so that users can make an informed decision but please in mind that the name field alone must be sufficient to discriminate between distinct test plans so you don’t have to duplicate that information in the description.

If your tests will require any special set-up (procuring external hardware, setting some devices or software in special test mode) it is recommended to include this information here.

The field has no size limit. It can contain newline characters. This field should be marked as translatable by prepending the underscore character (_) in front. This field is optional.

include:

A multi-line list of job identifiers, patterns matching such identifiers or template identifiers that should be included for execution.

This is the most important field in any test plan. It basically decides on which job definitions are selected by (included by) the test plan. Separate entries need to be placed on separate lines. White space does not separate entries as the id field may actually include spaces.

There are several options for selecting test cases:

  • You can simply list the identifier (either partial or fully qualified) of the job you want to include in the test plan directly. This is very common and most test plans used by Checkbox actually look like that.

  • You can also select many test cases at the same time, for instance to select jobs generated by template units, by:

    • using regular expressions. For example, storage-.* would select storage-sda, storage-sdb, etc. depending on the storage-{disk} template and the disks on the DUT.

    • using the template identifier. This will include all the jobs generated by this template.

Regardless if you use patterns or literal identifiers, you can use their fully qualified name (the one that includes the namespace they reside in) or an abbreviated form. The abbreviated form is applicable for jobs and templates that reside in the same namespace (but not necessarily the same provider) as the provider that is defining the test plan.

Checkbox will catch incorrect references to unknown jobs so you should be relatively safe. Have a look at the Examples section below for examples on how you can refer to jobs from other providers (you simply use their fully qualified name for that).

mandatory_include:

A multi-line list of job identifiers or patterns matching such identifiers that should always be executed.

This optional field can be used to specify the jobs that should always run. This is particularly useful for specifying jobs that gather vital info about the tested system, as it renders impossible to generate a report with no information about system under test.

For example, session results meant to be sent to the Ubuntu certification website must include the special job miscellanea/submission-resources:

mandatory_include:
    miscellanea/submission-resources

Note that mandatory jobs will always be run first (along with their dependent jobs).

bootstrap_include:

A multi-line list of job identifiers that should be run first, before the main body of testing begins. The job that should be included in the bootstrapping sections are the ones generating or helping to generate other jobs.

Example:

bootstrap_include:
    graphics/generator_driver_version

Note that each entry in the bootstrap_include section must be a valid job identifier and cannot be a regular expression pattern. Also note that only resource jobs are allowed in this section.

nested_part:

A multi-line list of test-plan identifiers whose contents will become part of this test-plan. This is a method of creating a tree of test plans, something that can be useful for organization and de-duplication of test plan definitions. For a full discussion of this capability see Checkbox nested test plans tutorial.

exclude:

A multi-line list of job identifiers or patterns matching such identifiers that should be excluded from execution.

This optional field can be used to prevent some jobs from being selected for execution.

This field may be used when a general (broad) selection is somehow made by the include field and it must be trimmed down (for example, to prevent a specific dangerous job from running). It has the same syntax as the include.

When a job is both included and excluded, exclusion always takes priority.

category_overrides:

A multi-line list of category override statements.

This optional field can be used to alter the natural job definition category association. Currently Checkbox allows each job definition to associate itself with at most one category (see Category Unit and Job Unit for details). This is sub-optimal as some tests can be easily assigned equally well to two categories at the same time.

For that reason, it may be necessary, in a particular test plan, to override the natural category association with one that more correctly reflects the purpose of a specific job definition in the context of a specific test plan.

For example let’s consider a job definition that tests if a specific piece of hardware works correctly after a suspend-resume cycle. Let’s assume that the job definition has a natural association with the category describing such hardware devices. In one test plan, this test will be associated with the hardware-specific category (using the natural association). In a special suspend-resume test plan the same job definition can be associated with a special suspend-resume category.

The actual rules as to when to use category overrides and how to assign a natural category to a specific test is not documented here. We believe that each project should come up with a workflow and semantics that best match its users.

The syntax of this field is a list of statements defined on separate lines. Each override statement has the following form:

apply CATEGORY-IDENTIFIER to JOB-DEFINITION-PATTERN

Both apply and to are literal strings. CATEGORY-IDENTIFIER is the identifier of a category unit. The JOB-DEFINITION-PATTERN has the same syntax as the include field does. That is, it can be either a simple string or a regular expression that is being compared to identifiers of all the known job definitions. The pattern can be either partially or fully qualified. That is, it may or may not include the namespace component of the job definition identifier.

Overrides are applied in order and the last applied override is the effective override in a given test plan. For example, given the following two overrides:

apply cat-1 to .*
apply cat-2 to foo

The job definition with the partial identifier foo will be associated with the cat-2 category.

certification_status_overrides:

A multi-line list of certification status override statements.

Similar to the category_overrides field, this field can be used to modify the certification status of the jobs matching the given regular expression.

The possible values are the same as for the certification-status job field.

For instance, the following will force every wireless job to become a certification blocker:

apply blocker to .*wireless.*

Note

This override mechanism “bubbles up”, meaning if it is used in a nested part, all the jobs selected as part of the whole test plan will be impacted, not only the jobs in that specific nested part.

Warning

If no namespace is provided, Checkbox will add the namespace of the current provider to the regular expression that is provided. This can be a problem if the override has to work across different providers with different namespaces. In that case, instead of using:

apply blocker to .*

the following should be used:

apply blocker to .*::.*

to apply the override to every job of every namespaces.

estimated_duration:

An approximate time to execute this test plan, in seconds.

This field can be expressed in two formats. The old format, a floating point number of seconds is somewhat difficult to read for larger values. To avoid mistakes test designers can use the new format with separate sections for number of hours, minutes and seconds. The format, as regular expression, is (\d+h)?[: ]*(\d+m?)[: ]*(\d+s)?. The regular expression expresses an optional number of hours, followed by the h character, followed by any number of spaces or : characters, followed by an optional number of minutes, followed by the m character, again followed by any number of spaces or : characters, followed by the number of seconds, ultimately followed by the s character.

The values can no longer be fractional (you cannot say 2.5m you need to say 2m 30s). We feel that sub-second granularity does is too unpredictable to be useful so that will not be supported in the future.

This field is optional. If it is missing it is automatically computed by the identical field that may be specified on particular job definitions.

Since sometimes it is easier to think in terms of test plans (they are typically executed more often than a specific job definition) this estimate may be more accurate as it doesn’t include the accumulated sum of mis-estimates from all of the job definitions selected by a particular test plan.

Examples

A simple test plan that selects several jobs:

id: foo-bar-and-froz
_name: Tests Foo, Bar and Froz
_description:
    This example test plan selects the following three jobs:
        - Foo
        - Bar
        - Froz
include:
    foo
    bar
    froz

A test plan that uses jobs from another provider’s namespace in addition to some of its own definitions:

id: extended-tests
_name: Extended Storage Tests (By Corp Inc.)
_description:
    This test plan runs an extended set of storage tests, customized
    by the Corp Inc. corporation. In addition to the standard Ubuntu
    set of storage tests, this test plan includes the following tests::

    - Multipath I/O Tests
    - Degraded Array Recovery Tests
include:
    com.canonical.certification:disk/.*
    multipath-io
    degrade-array-recovery

A test plan that generates jobs using the bootstrap_include section:

unit: test plan
id: test-plan-with-bootstrapping
_name: Tests with a bootstrapping stage
_description:
    This test plan uses the `bootstrap_include` field to generate
    additional jobs depending on the output of the `generator` job. The
    `include` section points to a template id, so Checkbox will run the
    jobs generated by this template, which are:
        - generated_job_Foo
        - generated_job_Bar
bootstrap_include:
    generator
include:
    generated_jobs

unit: job
id: generator
plugin: resource
_description: Job that generates Foo and Bar resources
command:
 echo "my_resource: Foo"
 echo
 echo "my_resource: Bar"

unit: template
template-unit: job
template-resource: generator
template-id: generated_jobs
plugin: shell
estimated_duration: 1
id: generated_job_{my_resource}
command: echo {my_resource}
_description: Job instantiated from template that echoes {my_resource}

Note

Although the method above is recommended, the same result can be obtained by modifying the include section of the test plan to use the regular expression generated_job_.* instead of the template id generated_jobs:

unit: test plan
id: test-plan-with-bootstrapping
_name: Tests with a bootstrapping stage
_description:
    This test plan uses bootstrapping_include field to generate additional
    jobs depending on the output of the generator job.
include:
    generated_job_.*
bootstrap_include:
    generator

A test plan that marks some jobs as mandatory:

unit: test plan
id: test-plan-with-mandatory-jobs
_name: Test plan with mandatory jobs
_description:
    This test plan runs some jobs regardless of user selection.
include:
    Foo
mandatory_include:
    Bar

unit: job
id: Foo
_name: Foo job
_description: Job that might be deselected by the user
plugin: shell
command: echo Foo job

unit: job
id: Bar
_name: Bar job (mandatory)
_description: Job that should *always* run
plugin: shell
command: echo Bar job