.. _test-plan: ============== Test Plan Unit ============== The test plan unit is a facility that describes a sequence of :ref:`jobs ` or :ref:`templates ` that should be executed together. Jobs are selected by either listing their identifier or a regular expression that matches their identifier. Listing a template identifier will select all the jobs instantiated by it. Selected jobs are executed in the sequence they appear in the list, unless they need to be reordered to satisfy dependencies which always take priority. It is also possible to :ref:`exclude` jobs from the selection using the same principles. Test plans can be :ref:`nested`, so you can define several smaller test plans and combine them into a bigger one, which helps with maintenance and flexibility. Test plans can contain additional meta-data which can be used in a graphical user interface. You can assign a translatable name and description to each test plan. Test plans are also typical units so they can be defined with the familiar :ref:`RFC822-like syntax` that is also used for job definitions. There can also be multiple test plan definitions per file, just like with all the other units, including job definitions. Test Plan Fields ================ The following fields can be used in a test plan. Note that **not all** fields need to be used or even should be used. Please remember that Checkbox needs to maintain backwards compatibility so some of the test plans it defines may have non-typical constructs required to ensure proper behavior. You don't have to copy such constructs when working on a new test plan from scratch .. _Test Plan id field: ``id``: Each test plan needs to have a unique identifier. This is exactly the same as with other units that have an identifier (like job definitions and categories). This field is not used for display purposes but you may need to refer to it on command line so keeping it descriptive is useful .. _Test Plan name field: ``name``: A human-readable name of the test plan. The name should be relatively short as it may be used to display a list of test plans to the test operator. Remember that the user or the test operator may not always be familiar with the scope of testing that you are focusing on. Also consider that multiple test providers may be always installed at the same time. The translated version of the name (and icon, see below) is the only thing that needs to allow the test operator to pick the right test plan. Please use short and concrete names like: - "Storage Device Certification Tests" - "Ubuntu Core Application's Clock Acceptance Tests" - "Default Ubuntu Hardware Certification Tests". The field has a soft limit of eighty characters. It cannot have multiple lines. This field should be marked as translatable by prepending the underscore character (\_) in front. This field is mandatory. .. _Test Plan description field: ``description``: A human-readable description of this test plan. Here you can include as many or few details as you'd like. Some applications may offer a way of viewing this data. In general it is recommended to include a description of what is being tested so that users can make an informed decision but please in mind that the ``name`` field alone must be sufficient to discriminate between distinct test plans so you don't have to duplicate that information in the description. If your tests will require any special set-up (procuring external hardware, setting some devices or software in special test mode) it is recommended to include this information here. The field has no size limit. It can contain newline characters. This field should be marked as translatable by prepending the underscore character (\_) in front. This field is optional. .. _Test Plan include field: ``include``: A multi-line list of job identifiers, patterns matching such identifiers or template identifiers that should be included for execution. This is the most important field in any test plan. It basically decides on which job definitions are selected by (included by) the test plan. Separate entries need to be placed on separate lines. White space does not separate entries as the ``id`` field may actually include spaces. There are several options for selecting test cases: - You can simply list the identifier (either partial or fully qualified) of the job you want to include in the test plan directly. This is very common and most test plans used by Checkbox actually look like that. - You can also select many test cases at the same time, for instance to select jobs generated by :ref:`template units`, by: - using regular expressions. For example, ``storage-.*`` would select ``storage-sda``, ``storage-sdb``, etc. depending on the ``storage-{disk}`` template and the disks on the :term:`DUT`. - using the template identifier. This will include all the jobs generated by this template. Regardless if you use patterns or literal identifiers, you can use their fully qualified name (the one that includes the :term:`namespace` they reside in) or an abbreviated form. The abbreviated form is applicable for jobs and templates that reside in the same namespace (but not necessarily the same provider) as the provider that is defining the test plan. Checkbox will catch incorrect references to unknown jobs so you should be relatively safe. Have a look at the :ref:`test-plan-examples` section below for examples on how you can refer to jobs from other providers (you simply use their fully qualified name for that). .. _Test Plan mandatory_include field: ``mandatory_include``: A multi-line list of job identifiers or patterns matching such identifiers that should always be executed. This optional field can be used to specify the jobs that should always run. This is particularly useful for specifying jobs that gather vital info about the tested system, as it renders impossible to generate a report with no information about system under test. For example, session results meant to be sent to the Ubuntu :term:`certification website` must include the special job ``miscellanea/submission-resources``:: mandatory_include: miscellanea/submission-resources Note that mandatory jobs will always be run first (along with their dependent jobs). .. _Test Plan bootstrap_include field: ``bootstrap_include``: A multi-line list of job identifiers that should be run first, before the main body of testing begins. The job that should be included in the bootstrapping sections are the ones generating or helping to generate other jobs. Example:: bootstrap_include: graphics/generator_driver_version Note that each entry in the bootstrap_include section must be a valid job identifier and cannot be a regular expression pattern. Also note that only resource jobs are allowed in this section. .. _Test Plan nested_part field: ``nested_part``: A multi-line list of test-plan identifiers whose contents will become part of this test-plan. This is a method of creating a tree of test plans, something that can be useful for organization and de-duplication of test plan definitions. For a full discussion of this capability see :ref:`nested-test-plan`. .. _Test Plan exclude field: ``exclude``: A multi-line list of job identifiers or patterns matching such identifiers that should be excluded from execution. This optional field can be used to prevent some jobs from being selected for execution. This field may be used when a general (broad) selection is somehow made by the ``include`` field and it must be trimmed down (for example, to prevent a specific dangerous job from running). It has the same syntax as the ``include``. When a job is both included and excluded, exclusion always takes priority. .. _Test Plan category-overrides field: ``category-overrides``: A multi-line list of category override statements. This optional field can be used to alter the natural job definition category association. Currently Checkbox allows each job definition to associate itself with at most one category (see :doc:`category` and :doc:`job` for details). This is sub-optimal as some tests can be easily assigned equally well to two categories at the same time. For that reason, it may be necessary, in a particular test plan, to override the natural category association with one that more correctly reflects the purpose of a specific job definition in the context of a specific test plan. For example let's consider a job definition that tests if a specific piece of hardware works correctly after a suspend-resume cycle. Let's assume that the job definition has a natural association with the category describing such hardware devices. In one test plan, this test will be associated with the hardware-specific category (using the natural association). In a special suspend-resume test plan the same job definition can be associated with a special suspend-resume category. The actual rules as to when to use category overrides and how to assign a natural category to a specific test is not documented here. We believe that each project should come up with a workflow and semantics that best match its users. The syntax of this field is a list of statements defined on separate lines. Each override statement has the following form:: apply CATEGORY-IDENTIFIER to JOB-DEFINITION-PATTERN Both 'apply' and 'to' are literal strings. CATEGORY-IDENTIFIER is the identifier of a category unit. The JOB-DEFINITION-PATTERN has the same syntax as the ``include`` field does. That is, it can be either a simple string or a regular expression that is being compared to identifiers of all the known job definitions. The pattern can be either partially or fully qualified. That is, it may or may not include the namespace component of the job definition identifier. Overrides are applied in order and the last applied override is the effective override in a given test plan. For example, given the following two overrides:: apply cat-1 to .* apply cat-2 to foo The job definition with the partial identifier ``foo`` will be associated with the ``cat-2`` category. .. _Test Plan estimated_duration field: ``estimated_duration``: An approximate time to execute this test plan, in seconds. This field can be expressed in two formats. The old format, a floating point number of seconds is somewhat difficult to read for larger values. To avoid mistakes test designers can use the new format with separate sections for number of hours, minutes and seconds. The format, as regular expression, is ``(\d+h)?[: ]*(\d+m?)[: ]*(\d+s)?``. The regular expression expresses an optional number of hours, followed by the ``h`` character, followed by any number of spaces or ``:`` characters, followed by an optional number of minutes, followed by the ``m`` character, again followed by any number of spaces or ``:`` characters, followed by the number of seconds, ultimately followed by the ``s`` character. The values can no longer be fractional (you cannot say ``2.5m`` you need to say ``2m 30s``). We feel that sub-second granularity does is too unpredictable to be useful so that will not be supported in the future. This field is optional. If it is missing it is automatically computed by the identical field that may be specified on particular job definitions. Since sometimes it is easier to think in terms of test plans (they are typically executed more often than a specific job definition) this estimate may be more accurate as it doesn't include the accumulated sum of mis-estimates from all of the job definitions selected by a particular test plan. .. _test-plan-examples: Examples ======== A simple test plan that selects several jobs:: id: foo-bar-and-froz _name: Tests Foo, Bar and Froz _description: This example test plan selects the following three jobs: - Foo - Bar - Froz include: foo bar froz A test plan that uses jobs from another provider's namespace in addition to some of its own definitions:: id: extended-tests _name: Extended Storage Tests (By Corp Inc.) _description: This test plan runs an extended set of storage tests, customized by the Corp Inc. corporation. In addition to the standard Ubuntu set of storage tests, this test plan includes the following tests:: - Multipath I/O Tests - Degraded Array Recovery Tests include: com.canonical.certification:disk/.* multipath-io degrade-array-recovery A test plan that generates jobs using the ``bootstrap_include`` section:: unit: test plan id: test-plan-with-bootstrapping _name: Tests with a bootstrapping stage _description: This test plan uses the `bootstrap_include` field to generate additional jobs depending on the output of the `generator` job. The `include` section points to a template id, so Checkbox will run the jobs generated by this template, which are: - generated_job_Foo - generated_job_Bar bootstrap_include: generator include: generated_jobs unit: job id: generator plugin: resource _description: Job that generates Foo and Bar resources command: echo "my_resource: Foo" echo echo "my_resource: Bar" unit: template template-unit: job template-resource: generator template-id: generated_jobs plugin: shell estimated_duration: 1 id: generated_job_{my_resource} command: echo {my_resource} _description: Job instantiated from template that echoes {my_resource} .. note:: Although the method above is recommended, the same result can be obtained by modifying the ``include`` section of the test plan to use the regular expression ``generated_job_.*`` instead of the template id ``generated_jobs``:: unit: test plan id: test-plan-with-bootstrapping _name: Tests with a bootstrapping stage _description: This test plan uses bootstrapping_include field to generate additional jobs depending on the output of the generator job. include: generated_job_.* bootstrap_include: generator A test plan that marks some jobs as mandatory:: unit: test plan id: test-plan-with-mandatory-jobs _name: Test plan with mandatory jobs _description: This test plan runs some jobs regardless of user selection. include: Foo mandatory_include: Bar unit: job id: Foo _name: Foo job _description: Job that might be deselected by the user plugin: shell command: echo Foo job unit: job id: Bar _name: Bar job (mandatory) _description: Job that should *always* run plugin: shell command: echo Bar job