.. _base_tutorial_commands: ============================ A few more Checkbox commands ============================ Checkbox comes with a bunch of commands. To see them, run the following in a terminal: .. code-block:: none checkbox.checkbox-cli --help usage: checkbox-cli [-h] [-v] [--debug] [--clear-cache] [--clear-old-sessions] [--version] {check-config,launcher,list,run,startprovider,submit,show, list-bootstrapped,merge-reports,merge-submissions,tp-export, service,remote} positional arguments: {check-config,launcher,list,run,startprovider,submit,show,list-bootstrapped, merge-reports,merge-submissions,tp-export,service,remote} subcommand to run options: -h, --help show this help message and exit -v, --verbose print more logging from checkbox --debug print debug messages from checkbox --clear-cache remove cached results from the system --clear-old-sessions remove previous sessions' data --version show program's version information and exit You've already seen the ``check-config`` and the ``launcher`` commands. In this section, you will use a few other commands that can be helpful to explore what's available in Checkbox. List available objects ====================== Checkbox handles a lot of "objects" when it's running. These objects can be the different Checkbox unit types (:ref:`jobs`, :ref:`test plans`, etc.), but they also include the test scripts, the test providers themselves and even the test sessions known to Checkbox! To get a list of all the available Checkbox objects on your system, run: .. code-block:: none checkbox.checkbox-cli list service 'service object' children provider 'com.canonical.plainbox:manifest' children file '/snap/checkbox22/current/lib/python3.10/site-packages/plainbox/impl/providers/manifest/README.md' file '/snap/checkbox22/current/lib/python3.10/site-packages/plainbox/impl/providers/manifest/bin/plainbox-manifest-collect' (...) job 'com.canonical.certification::tpm2.0_4.1.1/tpm2_verifysignature' job 'com.canonical.certification::tpm2.0_4.1.1/pcr0_mismatch_check' job 'com.canonical.certification::tpm2.0_4.1.1/context_gap_max_check' file '/snap/checkbox22/current/providers/checkbox-provider-tpm2/units/tpm2_4.1.1.pxu' That's **a lot** of objects! Fortunately, it is possible to retrieve objects from a specific group. Let's see what test plans are available, for instance: .. code-block:: none checkbox.checkbox-cli list "test plan" test plan 'com.canonical.certification::6lowpan-automated' test plan 'com.canonical.certification::acpi-automated' test plan 'com.canonical.certification::audio-cert-full' (...) test plan 'com.canonical.certification::tpm-cert-focal-automated' test plan 'com.canonical.certification::tpm-cert-automated' Combined with a ``grep`` command, it is possible to find all the test plans related to audio testing: .. code-block:: none checkbox.checkbox-cli list "test plan" | grep audio test plan 'com.canonical.certification::audio-cert-full' test plan 'com.canonical.certification::audio-cert-manual' (...) test plan 'com.canonical.certification::after-suspend-audio-pa-manual' test plan 'com.canonical.certification::after-suspend-audio-automated' Or let's list all the available jobs (test cases): .. code-block:: none checkbox.checkbox-cli list all-jobs id: com.canonical.certification::6lowpan/kconfig kernel config options for 6LoWPAN id: com.canonical.certification::IEEE_80211 Creates resource info for wifi supported protocols/interfaces id: com.canonical.certification::acpi/oem_osi test ACPI OEM _OSI strings (...) By default, the output for the ``all-jobs`` option is to list the job identifier followed by its summary (or ```` if there is no summary). We can tailor the output using the ``--format`` parameter and all the fields available from the jobs. To see what fields are available, run: .. code-block:: none checkbox.checkbox-cli list all-jobs --format ? Available fields are: _description, _purpose, _siblings, _steps, _summary, _template-summary, _verification, after, before, category_id, command, depends, description, environ, estimate_duration, estimated_duration, flags, full_id, id, imports, plugin, purpose, require, requires, steps, summary, template-engine, template-filter, template-id, template-resource, template-unit, unit, user, verification .. note:: These fields are explained in the :ref:`job` page. .. note:: The underscore before some of the fields names simply means the content of this field can be translated into another language. .. note:: There is currently a bug with this command. It does not actually list all the available fields. It shows only the fields that are used throughout the jobs that are in scope. In most cases, this will be all the fields available, but a new field that is not used in any job will not be listed. To create a table listing each job id and their summary, run: .. code-block:: none checkbox.checkbox-cli list all-jobs --format "{id:30}\t|\t{_summary}\n" 6lowpan/kconfig | kernel config options for 6LoWPAN IEEE_80211 | Creates resource info for wifi supported protocols/interfaces acpi/oem_osi | test ACPI OEM _OSI strings acpi_sleep_attachment | (...) xinput | Creates resource info from xinput output. collect-manifest | Collect the hardware manifest (interactively) manifest | Hardware Manifest .. note:: ``\n`` and ``\t`` in the formatting string are interpreted and replaced with new line and tab respectively. When using your own formatting, the jobs are not suffixed with a new line: you have to explicitly use it. List the content of a test plan as executed by Checkbox ======================================================= In the previous section, you've listed all the test plans related to audio. Select one of them, for instance ``com.canonical.certification::audio-cert-automated``, and see what jobs it contains: .. code-block:: none checkbox.checkbox-cli list-bootstrapped audio-cert-automated com.canonical.plainbox::manifest com.canonical.certification::package com.canonical.certification::audio/detect_sinks com.canonical.certification::device com.canonical.certification::audio/detect_sources com.canonical.certification::audio/alsa_record_playback_automated com.canonical.certification::audio/alsa_info_collect com.canonical.certification::audio/alsa_info_attachment com.canonical.certification::audio/list_devices com.canonical.certification::audio/valid-sof-firmware-sig .. note:: You can omit the namespace (that would be ``com.canonical.certification`` in this case from any of the following commands). When doing so, Checkbox will do its best to find what you are asking for, disregarding the namespace. If the match is not unique (there is more than one unit matching), the command may either fail (if it wouldn't make sense to apply the same action to all matches) or run the action for all matches. If this is undesirable, you can use ``--exact`` to fail if nothing exactly matches your query. If you were to run this test plan with Checkbox, it would run these jobs in the order shown above. .. note:: The name of this command refers to the Checkbox :term:`bootstrapping` phase. But what are these jobs exactly? You can use the ``show`` command to see the content of a Checkbox object. Show the content of a Checkbox object ===================================== Have a look at the ``com.canonical.certification::audio/list_devices`` job listed in the ``com.canonical.certification::audio-cert-automated`` test plan above, for instance. What is it exactly? What does it contain? What is its **definition?** Use the ``show`` command to find out: .. code-block:: none checkbox.checkbox-cli show com.canonical.certification::audio/list_devices origin: /snap/checkbox22/current/providers/checkbox-provider-base/units/audio/jobs.pxu:1-9 plugin: shell category_id: com.canonical.plainbox::audio id: audio/list_devices estimated_duration: 1.0 requires: device.category == 'AUDIO' package.name == 'alsa-base' command: cat /proc/asound/cards _description: Test to detect audio devices The first line tells you that this job comes from lines 1 to 9 of the ``/snap/checkbox22/current/providers/checkbox-provider-base/units/audio/jobs.pxu`` file. The other lines show its definition. We can see what it does ("Test to detect audio devices"), how (using the ``cat /proc/asound/cards`` command), and many other details. .. note:: Each of the fields shown in a job definition are explained in the :ref:`job` page. Now that you know the definition of this job, you can run it to see what the output generated by Checkbox look like. .. _run_subcmd: Run a particular test plan or a set of jobs =========================================== Run the following command: .. code-block:: none checkbox.checkbox-cli run ".*audio/list_devices" ===========================[ Running Selected Jobs ]============================ ==============[ Running job 1 / 3. Estimated time left: 0:00:03 ]=============== -----------[ Collect information about installed software packages ]------------ ID: com.canonical.certification::package Category: com.canonical.plainbox::uncategorised ... 8< ------------------------------------------------------------------------- name: accountsservice version: 22.07.5-2ubuntu1.4 (...) ==============[ Running job 3 / 3. Estimated time left: 0:00:01 ]=============== -----------------------------[ audio/list_devices ]----------------------------- ID: com.canonical.certification::audio/list_devices Category: com.canonical.plainbox::audio ... 8< ------------------------------------------------------------------------- 0 [PCH ]: HDA-Intel - HDA Intel PCH HDA Intel PCH at 0xf7f10000 irq 31 1 [H340 ]: USB-Audio - Logi USB Headset H340 Logitech Inc. Logi USB Headset H340 at usb-0000:00:14.0-1, full speed 2 [HDMI ]: HDA-Intel - HDA ATI HDMI HDA ATI HDMI at 0xf7e60000 irq 32 3 [U0x46d0x802 ]: USB-Audio - USB Device 0x46d:0x802 USB Device 0x46d:0x802 at usb-0000:00:14.0-4, high speed ------------------------------------------------------------------------- >8 --- Outcome: job passed Finalizing session that hasn't been submitted anywhere: checkbox-run-2023-07-24T09.01.24 ==================================[ Results ]=================================== ☑ : Collect information about installed software packages ☑ : Collect information about hardware devices (udev) ☑ : audio/list_devices A few things to notice: - Checkbox has executed the ``com.canonical.certification::audio/list_devices`` job. Because it understands regular expression patterns, you were able to pass it ``.*audio/list_devices`` instead of the full job id. - The output is quite similar to running Checkbox manually, except only a text summary is generated at the end (no submission files, no request to upload results to the Certification website). - Although you asked Checkbox to run one job, it actually ran three. This is because the ``audio/list_devices`` job definition has some requirements from other jobs (namely, ``com.canonical.certification::packages`` and ``com.canonical.certification::device``), so these jobs are executed as well when you run it. You can execute a set of jobs by using an appropriate regular expression. For instance, ``checkbox.checkbox-cli run .*audio/.*`` would run every job whose ``id`` contain the string ``audio/`` (as well as the jobs they depend on). .. warning:: Be careful when using the ``run`` command with such open regular expressions because you might end up running quite a lot of jobs! You can also run a whole test plan using the ``run`` command: .. code-block:: none checkbox.checkbox-cli run com.canonical.certification::tutorial-base This will run the Checkbox Base Tutorial test plan, executing all the jobs in it and providing a text summary of the test run. Getting an example launcher =========================== It is often hard to track which section, keys and values are allowed in a launcher. Sometimes it is also tricky to remember what the defaults are for each configuration to decide if it is useful to override it in this case or not. Try to run the following command: .. code-block:: checkbox.checkbox-cli config defaults [launcher] # Version of launcher to use # type: int launcher_version = 1 # Identifier of the application # type: str app_id = checkbox-cli # Version of the application # type: str app_version = # List of stock reports to use # type: list stock_reports = ['text', 'certification', 'submission_files'] # Send/generate submission report locally when using checkbox remote # type: bool local_submission = True # A title to be applied to the sessions created using this launcher that can be used in report generation # type: str session_title = session title # A string that can be applied to sessions created using this launcher. Useful for storing some contextual infomation about the session # type: str session_desc = [test plan] # Constrain interactive choice to test plans matching this glob # type: list filter = ['*'] # Select this test plan by default. # type: str unit = # Don't allow the user to change test plan. # type: bool forced = False [...] As you can see, this command prints a default launcher you can redirect to a file and use out of the box, pre-populated with all the default values and comments to explain what each configuration means. Wrapping up =========== Congratulations! You've completed the Checkbox usage tutorial! You've installed Checkbox, run a test plan and reviewed the report generated, written a launcher to customize a test run, discovered how to run Checkbox on one device to control another device, and used Checkbox commands to navigate the available objects in Checkbox. You are now ready to use Checkbox to test any kind of devices! In the :ref:`Writing test jobs tutorial `, you will learn how to write new tests for Checkbox and how to create your own Checkbox-based snaps.