Canary Testing for Checkbox Edge Version: In-depth Process¶

Introduction¶

Canary testing applies on Checkbox snaps that are released through the edge channel in the Snap Store. New versions are built daily if changes are made in the code repository.

The following sections provide a detailed walk-through of the Canary Testing process for the Checkbox Edge version, starting from snap building to the outcomes of the testing.

Snap Build and Release¶

Trigger Conditions¶

The GitHub action responsible for building the snap runs every day. However, it only triggers when the following condition is met:

At least one pull request (PR) has been merged since the last edge build.

Build Workflow¶

To monitor the build process, or to review the configuration and logs, see the GitHub workflow.

Post-Build Actions¶

Once the build is successful, the snap packages are automatically pushed to the Snap Store in the edge channel. Testers and early adopters can access the latest version through edge releases.

Jenkins Monitoring and Validation¶

Snap Monitoring¶

Once the snap is published to the edge channel in the Snap Store, our Jenkins job titled checkbox-edge-validation-detect-new-build gets into action.

Monitoring job URL¶

This Jenkins job monitors the Snap Store for the presence of the new snap using the following URL:

https://api.snapcraft.io/v2/snaps/find?q=checkbox22&channel=edge&fields=revision&architecture=amd64

The specific JSON path that’s being monitored for changes is:

$.results[0].revision.revision

The job checks this path every minute for updates.

Validation and Wait Logic¶

Upon detecting a new snap:

The job verifies the presence of all other related checkbox snaps.
If any snap is missing, the job waits for an hour, periodically checking for its availability.
If the snaps are available within the waiting period, the next stage of testing is initiated.

checkbox-edge-canary-validation Pipeline¶

Upon successful snap validation, the checkbox-edge-canary-validation pipeline begins its operation.

The Canary Pipeline contains the groovy script implementing the pipeline.

Testing Platforms and Specifications¶

The pipeline concurrently runs 3 distinct jobs on different platforms:

amd64 ubuntu core 22: Utilizing machines that consume Testflinger tasks from the dearest team queue.
amd64 ubuntu core 16: Also using machines from the dearest team queue, which corresponds to generic x86_64 machines.
arm64 ubuntu core 22: Targeting machines in the cert-rpi4b4g queue, which corresponds to the Raspberry Pi4 4GB model.

The Example of a validation job contains the Jenkins job definition for the amd64 ubuntu core 22 validation.

For a detailed look on how the job execution is carried out by all the entities in the chain, refer to the Validation pipeline execution.

Canary Test Plan Criteria¶

The canary test plan outlines specific tests that are imperative for the new snap’s validation. The pipeline’s successful conclusion is contingent upon all these tests passing on each of the mentioned platforms.

Outcome and Information propagation¶

Should the validation complete without errors, the beta reference in the Checkbox repository is set to point at the revision that got validated. The beta branch head is then updated on GitHub.

Conclusion¶

This Canary Testing process, complete from snap building to testing, ensures that every release of Checkbox in the edge channel is thoroughly vetted and stable.