TrustedFirmware OpenCI Users Guide

Table of Contents

Table of contents generated with markdown-toc

User Guide

This Google Document is a draft for development and review. Once reviewed, the doc will be made available on Phabricator on the website where additional content will be added and reviewed.

This guide briefly explains how to use and contribute to the Trusted Firmware project, in particular the Trusted Firmware A and Trusted Firmware M

How to Contribute to this Document

As a community project, help in all areas is greatly appreciated. If while using this document, issues are found or content can be improved, please follow the steps in this section to add your improvements to this document:

Note the steps were validated using Ubuntu 20.4. Other versions/OS's may need to be adjusted for.

You are now ready to set up your local environment

sudo apt-get update
sudo apt install git
sudo apt install git-review

(This may install any required dependencies automatically.)

How to Contribute Code

The Trusted Firmware core projects, TF-M and TF-A, are both open source projects and both share the same way to accept user contributions. Code changes, commonly known as patches or patchsets, are git-tracked so once the corresponding project is cloned, all content (history of previous patches which now are commits) is available at the user's machine.

TF projects use Gerrit as a centralized system to push, update, review and review patches. Contributors must push their patches to gerrit, so CI and reviewers can see the proposed change. One can see the open ones at

alt text

Once a patch is approved, one of the core maintainers merges it to the main branch (master) through Gerrit. The same cycle is repeated for every patch, where a patch series may be merged in one step.

Mailing lists are used to communicate latest news and also it is the main channel for users to post questions or issues, so it is a good idea to subscribe to these. Note, the mailing lists are not intended for patch reviews, so patches should go into Gerrit and news/questions/issues through the mailing lists.

Gerrit Setup

Once the project is cloned, there are some two extra steps to setup gerrit properly: 1. setting up the gerrit remote repository and 2. Git-review package installation

For example, under the TF-A project, use the following command to add the remote

$ git remote add gerrit ssh://<gerrit user>

For TF-M, the same command applies except that remote’s url is a bit different

$ git remote add gerrit ssh://<gerrit user>

As a safety check, run the command git remote -v and make sure gerrit remote is present. The next step is to install the git-review package in your corresponding Linux distribution. For example, on Ubuntu this would be

$ sudo apt install git-review

and finally define the gerrit remote to be used with the following command

$ git review -r

If no issues are found at this point, you should be ready to start contributing to the project!

Commit and review

No matter what change you want to make in any repository, one needs to create one or more commits into a local branch before submission. All commits must have ‘Signed-off-by’ and ‘Change-id’ strings in the commit description otherwise submission fails. The ‘Signed-off-by’ is introduced explicitly by the user (git commit -s) and the ‘Change-id’ automatically created by the git-review plugin. Patches should be atomic, just targeting one task. A commit’s subject should answer the question ‘what changed’ and the commit’s description answers the question ‘why it changed’. Be clear and always use present verbs, i.e use Add instead of Adding.

Once your commits are ready, type

git review

This command takes care of all the internal commands needed to send the patch to Gerrit, as seen below:

alt text

Once a patch is submitted, you must include one or more reviewers. The question then raises: who should I add as a reviewer? One simple approach would be to look at the git history of the files you are modifying, and look for authors who have committed recently.

git log <path to file>

Once reviewers are included, you would probably get some feedback pretty soon. TF projects are quite active but in case you do not get any activity in a couple of days, reply from Gerrit indicating that you would like some feedback. Take the time to understand and review every comment and response properly, do corrections and update the patch promptly if required. Keep polishing the patch until all feedback/observations are resolved. A good practice is to create a new branch for each patch update (suffix a version number on the branch name) so one can switch back and forth between patch versions. CI output and reviewers’ comments are reflected in gerrit and email (the one that appears in your patch metadata), so be sure to check any of these after submission.

More details about Gerrit can be found in the upstream documentation:

From the maintainer's POV

All CI is done with Jenkins at There are lots of jobs, so as a first impression, it may seem difficult to follow the CI flow. Section 4, Pipeline description, describes each project’s CI in detail.

alt text

Each project, TF-A and TF-M, have different CI jobs and scripts hosted in the following repos

The job config repositories contain Jenkins Job Definitions, called JJB (Jenkins Job Builders) files. The CI scripts repositories host scripts that are required for the CI, i.e build scripts, static checks, etc.

Manual Job trigger

For patches that arrive at gerrit, the CI is explicitly triggered by a core maintainer. However there are cases where a particular job needs to be rebuilt. Jobs can be rebuilt at any level, from the trigger job to the job that builds or launches the LAVA execution. For example, the below picture shows a trigger job with the ‘Build with Parameters’ and ‘Rebuild last’ options. Both options allow the maintainer to change any job parameter before actually executing it.

Most probably, you may want to go to a specific failed job and ‘Rebuild’

There may be many reasons to rebuild but perhaps the most trivial one is to make sure the error is valid and not a transient one. Look at the job’s console for errors.

LAVA documentation

LAVA(Linaro Automation and Validation Architecture) provides detailed documentation that can be found here.

Supported platforms

TF project support the following platforms in LAVA

alt text

TF-A supported platforms

Until recently, the only supported platform was the Juno board but now it also supports FVP models:

FVP models are virtual platforms that are able to emulate specific Arm reference designs or platforms. See the Arm documentation for more details:

These models are present on several docker images and LAVA uses these to boot and test a particular model. The specific docker image and model type are defined in the (LAVA) job definition. Docker images (containing the models) are available in a private docker registry ( that LAVA has access to. At the time of this writing, these are the docker image names

Tag name, i.e. fvp_base_revc-2xaemv8a_11.12_38, corresponds to the particular model download from

TF-M supported platforms

TF-M LAVA devices are:

How to navigate

All LAVA jobs triggered by Jenkins are executed at the TF LAVA lab instance. The Jenkins jobs that launch LAVA jobs are: tf-a-builder and tf-m-lava-submit The Jenkins jobs contain the corresponding LAVA id which can be used to find the corresponding job at In case of tf-a-builder job, the LAVA log itself is fetched from LAVA lab and attached to the job as seen below:

alt text

At the link, one can go to a particular job to check results directly from LAVA:

alt text

One useful feature is the possibility to resubmit jobs: this enables the user to quickly modify a job definition and test it without the need to retrigger the full build from Jenkins:

How to read a job and investigate results

The entrypoint to investigate a job failure is looking at the job’s landing page, i.e.

alt text

Depending on the device type and job definition, the output can vary considerably. One can filter out relevant logs by clicking the different log levels. As in any system, failures can occur at any time and for different reasons, i.e introduced by a user's patch or scripts/infrastructure. In any case, a core maintainer should monitor and report or fix it accordingly.

Pipeline description

The TF Open CI is divided into two separate pipelines, each handling the respective project (TF-A vs TF-M). Each is different in design so we describe each separately.

TF-A CI pipeline description

The TF-A CI pipeline underwent refactoring based on requirements in, going from testing a single test configuration to hundreds of them. Besides improving considerably the QA, it added some complexity as we will see below.

At the time of this writing, there are two (mostly) identical CIs, one running inside Arm (Internal CI) and one at (Open CI). These are two CI instances running in parallel but in different environments: the internal CI runs in a single node (master node) while the Open CI in multiple nodes (docker nodes). In the near future, only the Open CI will be running once all the internal CI features are fully migrated.

Below is a general picture of the Trusted firmware A CI flow. Top boxes are Jenkins jobs, except those with .sh extensions.

Alt text

The first job, the trigger-job, can be any job defined below, each covering a set of platforms and build/run configurations through test groups (TEST_GROUPS), ultimately split as test descriptions (TEST_DESC). A test description is tested by tf-a-builder job and a LAVA job is launched once artifacts are ready to be consumed.

In terms of the trigger source, jobs can be classified by either gerrit or scheduled:

All the above jobs rely on downstream jobs:

Any job can be triggered manually by authorized users. Gerrit jobs are those triggered on behalf of gerrit actions, either ‘Allow +1’ or ‘Allow +2’, and track a particular project, either TF-A or TF-A tests The following screenshot shows an example of the tf-gerrit-tforg-l1 job at the Jenkins instance:

Alt text

The tf-gerrit-tforg-l1 indicates the tracking project, tf-gerrit-tforg-l1, and the level, tf-gerrit-tforg-l1. Levels indicate testing depth (test descriptions coverage) and are used in different phases in the development phase as seen below:

Alt text

The job tf-daily runs daily, uses the latest code (the HEAD commit) and triggers two jobs: tf-main and tf-coverity. The job tf-main is the one covering most platforms so it takes longer to complete (approximately 1.5 hours)

Alt text

Notice that the tf-main job also triggers tf-static-checks, the job that launches project-related static checks (copyright presence, headers in alphabetical order, line endings, coding style and banned APIs) and execute Clang static analyzer (scan-build). The job tf-coverity runs the Coverity static code check and reports metrics (defects) at

The job tf-a-builder is the builder job and its execution is containerized inside docker-amd64-tf-a-bionic defined at repository. Anyone can fetch it with the following command and use it for local compilation

docker pull trustedfirmware/ci-amd64-ubuntu:bionic

The result of tf-a-builder is a set of artifacts: binaries, build log, environment files, etc.

In particular, if build produces a LAVA definition file, job.yaml, a LAVA job is launched through SQUAD Once the LAVA job finishes, jenkins fetches the log from LAVA and stores it in the corresponding jenkins job. It is worth mentioning that not all tf-a-builder jobs produce a LAVA job, i.e fvp model not supported, ‘nil’ run configuration provided in the test description, static check, etc. however most FVP and Juno produce one. One can see all executed LAVA jobs at

Finally, depending on the CI execution outcome, this is reflected in Gerrit as ‘TrustedFirmware Code Review’ comments:

Alt text

Results from those LAVA executed jobs on behalf of the corresponding gerrit patch are also reflected in Gerrit:

Alt text

In case of a job failure, it is more likely that you want to investigate the issue starting at the Gerrit job, then following the CI job chain starting from the trigger job (top-bottom approach): 1. analyze results from the Gerrit job, 2. use the report table produced by each tf-a-ci-gateway job, 3. the tf-a-builder job and finally 4. the tf-a-builder’s console log. Looking at the following screenshots should help clarifying this concept.

  1. Gerrit job level:

Alt text

  1. tf-a-ci-gateway level:

Alt text

  1. tf-a-builder level:

Alt text

  1. tf-a-builder’s console log view:

Alt text

TF-M CI pipeline description

TF-M jobs are found at and can be classified depending on the code coverage

Below is a diagram that shows their relationship and the amount of code coverage targeted.

Alt text

Jobs can also be classified depending on their specific task:

TF-M Job dependencies

When a patch arrives at and reviewed, a maintainer may allow the CI to be executed, which in turn triggers the tf-m-static job. This is exactly the same CI workflow as for TF-A. In case of failure, the patch cannot be merged into the stable branch. The tf-m-static triggers many more jobs as seen in the picture below

Alt text

The job tf-m-nigthly is a more extensive job, triggered everyday and tests the latest code (HEAD) at the project

Alt text

In case the nightly job fails, an email notification is sent through the mailing list . The maintainer is responsible for looking at the failed errors and identifying the (commit) culprit then reporting it to the developer.

The TF Jenkins Job Builder (JJB) configs

The TF project uses YAML files to define Jenkins jobs using Jenkins Job Builder (JJB): Jobs currently defined for both projects are at and Job triggers are special types of jobs that listen to certain gerrit events. For example the job triggers every time a TF-A maintainer ‘Allows +1’ the CI to execute as defined the job’s trigger section:

    - gerrit:   
            - comment-added-event:
            approval-category: "Allow-CI"
            approval-value: 1
        - project-compare-type: PLAIN
            project-pattern: TF-A/trusted-firmware-a
                - branch-compare-type: PLAIN
                branch-pattern: integration

JJBs and Jenkins Jobs

JJB defines the behaviour of a Job through a YAML file, where Jenkins use these to create jobs (it is similar to Class and Object concepts in Object Oriented Programming). For example this is JJB of TF-A Level1 trigger: which is instanciated at Similar pattern applies for the rest of the JJB files.

Calling CI scripts from JJB jobs

JJB files themselves do not do much unless they execute something useful. CI scripts are kept in separate repositories depending on the project. Below is the relationship between jobs and scripts repositories per project

In general, Jenkins jobs call scripts, the latter do the corresponding task. For example, below is shown again CI flow for the TF-A project

Alt text

Where is just a setup script (located at TF-A jobs repo) that finally calls script located CI scripts repo, which is the entrypoint of the script execution. The in turn calls others scripts that finally build the package.

CI Scripts overview

TF-A CI scripts overview

The TF-A CI repository contains several folders and scripts for different purposes but we will not describe each one. Instead we will overview build package operation. Building a package means building (compiling) a specific platform with certain build parameters and post-build setup tasks, both indicated in a single test configuration (string or filename). The operation is depicted in the following diagram:

Alt text

The test configuration concisely specifies a single test: what set of images to build, how to build them, and finally, how to run a test using the aforementioned images. A test configuration is a specially-named plain text file whose name comprises two parts: the build configuration and the run configuration.

The test configuration file is named in the following format:

{tf-build-config | nil}[,tftf-build-config]: { run-config | nil}

That is, it contains:

The TF and TFTF build configs are separated by a comma; the build and run configs are separated by a colon. The test configuration is consumed by the build script , and produces a build package. For example, the test configuration fvp-default,fvp-default:fvp-tftf-fip.tftf-aemv8a-debug chooses:

Build configurations are plain text files containing build parameters for a component; either TF or TFTF. The build parameters are sorted and listed one per line, and would appear on the component's build command line verbatim. Up to two build configurations can be specified – one for TF (mandatory), and another one for TFTF (optional). If the test doesn't require Trusted Firmware to be built (for example, for a TFTF build-only configuration), it must be specified as nil.

For example, the TF build config fvp-aarch32-tbb-mbedtls-rsa-ecdsa-with-ecdsa-rotpk-rsa-cert has the following contents as of this writing:


Build configs are located under tf_config and tftf_config subdirectories in the CI repository.

As described above, the build configuration describes what components to build, and how to build them. Before a set of images can be exercised through the test, the CI usually needs to execute a sequence of steps that are necessary to set up the test environment. These steps largely depend on the specific nature of the test at hand, the platform to be run on, etc. These steps are related to but decoupled from the build configs and are defined in run configurations. Almost all tests run in the CI mandatorily require a certain combination of steps above, some others optional. Because of the variability in applying the steps, and to avoid duplication, common steps are made available as standalone script snippets, called fragments. Individual fragments can be strung together to form a Run Configuration. Run config fragments are located under run_config subdirectory in the CI repository.

For example, the following test configuration


Produces the following build configs

Trusted Firmware config:


Trusted Firmware TF config:


And the following run config fragments


Producing the following (release) build package

├── artefacts
│   ├── build.log
│   ├── debug
│   │   ├── bl1.bin
│   │   ├── bl1.elf
│   │   ├── bl2.bin
│   │   ├── bl2.elf
│   │   ├── bl2u.bin
│   │   ├── bl2u.elf
│   │   ├── bl31.bin
│   │   ├── bl31.elf
│   │   ├── bl32.bin
│   │   ├── bl32.elf
│   │   ├── cactus.bin
│   │   ├── cactus.dtb
│   │   ├── cactus.elf
│   │   ├── cactus_mm.bin
│   │   ├── cactus_mm.elf
│   │   ├── el3_payload.bin
│   │   ├── fip.bin
│   │   ├── fvp-base-gicv3-psci.dtb
│   │   ├── fvp_fw_config.dtb
│   │   ├── fvp_nt_fw_config.dtb
│   │   ├── fvp_soc_fw_config.dtb
│   │   ├── fvp_tb_fw_config.dtb
│   │   ├── fvp_template.yaml
│   │   ├── fvp_tsp_fw_config.dtb
│   │   ├── fvp.yaml
│   │   ├── ivy.bin
│   │   ├── ivy.dtb
│   │   ├── ivy.elf
│   │   ├── job.yaml
│   │   ├── model_params
│   │   ├── ns_bl1u.bin
│   │   ├── ns_bl1u.elf
│   │   ├── ns_bl2u.bin
│   │   ├── ns_bl2u.elf
│   │   ├── quark.bin
│   │   ├── quark.dtb
│   │   ├── quark.elf
│   │   ├── run
│   │   ├── tftf.bin
│   │   └── tftf.elf
│   ├── env
│   └── release
├── fvp_template.yaml
├── fvp.yaml
├── job.yaml
├── lava_model_params
├── tmp.FlNca0PGGF
├── tmp.KMJFcZ0Zr6
├── tmp.ku5nXd85b4
├── tmp.mCaqKgvgfT
└── tmp.Sv3zjKIWz7

Ultimately, the job.yaml file above is the LAVA job definition, which contains the information required by LAVA (artefacts’ URL, model params, container containing the model, etc.) for a correct job execution.

TF-M CI scripts overview

**TODO - This section is missing**

TF LAVA Instance

The LAVA instance for the Trusted Firmware project is set up in Linaro Harston LAB. It consists of lava-master running on a hosted bare metal server, lava-dispatcher running on the same server. Additional dispatchers are deployed using Raspberry Pi 4 hardware. Note that this is required by certain classes of hardware not being differentiable before the OS boots, therefore a single device per dispatcher allows addressing these devices via an unique dispatcher (per device).

TF LAVA instance settings are stored in salt and ansible repositories:

TF LAVA instance replication

TF instance partially relies on Linaro infrastructure. Linaro’s login service (based on LDAP) is used for users authentication and logging into the TF LAVA instance. Therefore it’s not possible to replicate identical LAVA instance accounts outside of Linaro’s infrastructure. Apart from that, all configurations are stored in Salt or Ansible repositories. Replicating the remaining part of the instance can be done using Salt and Ansible tools with a new set of inventory variables.

Before an instance is ready various Ansible playbooks need to be run and, for LAVA setups, Salt needs to be run.

For Ansible, you need to go on deb-ansible host (ssh root@ As root:

    # (cd /srv/lava-lab; git pull)
    # cd /etc/ansible/playbooks
    # ansible-playbook -i ../inventory/tf lava-lab.yml 

The following playbooks are used to configure all the relevant parts:

Installing LAVA (worker and master) is a manual process. After that, the lava-lab.yml file takes care of setting up the correct device dictionaries, device types and health checks as configured in the separate lava-lab repository.

Until the Salt migration to Ansible is complete you will need to go on tf-master.tflab host (ssh root@ As root:

    # (cd /srv/lava-lab; git pull)
    # salt ‘*’ state.highstate

Note: on a brand new installation, you will need to run the salt command twice. It’s due to an ordering problem in the Salt state configuration. It will be fixed by the Ansible migration.

LAVA Master

LAVA Master and dispatchers run the Debian distribution (at the time of writing, Debian 10 Buster). LAVA packages are installed from repository. On top of the basic installation, LAB specific configuration is applied with Ansible. Note: the installation of lava-server is a manual process (and still a work in progress), while other configurations are automated and described in the Ansible playbooks above.

LAVA Dispatchers

TF instance uses 2 types of dispatchers:

LAVA dispatchers setup is described in the LAVA documentation:


Upgrades of LAVA software are performed after each LAVA release. All dispatchers and master have to run the same version of LAVA software.

LAVA instance changes

All the changes are done by the Linaro Lab staff. They should be requested as Jira tickets with the following fields:

Most common cases where ticket is required include:

Current list of available devices

Up-to-date list of devices is available from the LAVA web UI. A simplified view shows only the device types. Currently, TF LAVA instance has Juno, MPS2, Musca B1 and QEMU devices.

Local LAVA instance set up

Setting up a local LAVA instance that can be used for debugging or improving LAVA code, as well as new device enablement can be done in a few ways. The easiest is to use the official LAVA’s docker-compose repository and follow the README instructions.

New device enablement in LAVA

Enabling new devices in LAVA is described in the LAVA documentation.

Board setup


More details on Collaborate page:



After various iterations of deployment methods, the current method is loading a master image on SD card or USB stick, and booting a known good image from that. The known good image can be found here.


The most common issue with Juno is broken PDU ports. The ports get stuck in ON mode so the board never reboots and can thus not interrupt the boot loader. Another common issue is "Failed to erase old recovery image" which is generally an issue with the SD card. It is solved as follows:

  1. Take brand new SD card
  2. Use a root/sudo user to run following commands on your SD reader capable *nix device
  3. Run parted /dev/DISK_ID
  4. mklabel msdos
  5. mkpart
    1. primary
    2. fat16
    3. 1M
    4. 2G
  6. exit parted
  7. mkfs.fat16 /dev/DISK_ID -n JUNO<details>
  8. Download the recovery image from a health check
  9. Unpack and copy contents to SD card.
  10. Put a new card in a Juno device and run a health check. It might have umount issues on the first try. If so, try again.


More details in Collaborate page:



The technical reference manual can be found here.

An example health check with an image to use for deployment on MPS2 devices.

There is also access to a device dictionary which describes the process of using SDMux with the board.

In order to use SDMux, the host must have sd-mux-ctrl installed.


We found that the SD card containing the boot image easily got corrupted and that would take the board offline until manual intervention is made.

This is mitigated with use of the SDMux and there have been few known issues since.

SD Mux can be bought from

Musca B1

More details in Collaborate page:

Flashing the device for the first time

Flashing instructions are available on ARM community pages. It's only possible to run the Windows version of the instructions. Currently the Lab uses QSPI firmware version 3.4.

Firmware: DAPLink_QSPI_V34.bin

After initial flashing is done, the rest of the setup can be done with a Linux host. Some commands for DAPLink can be found in DAPLink docs.

Turn on automation

Boards need to have the "automation" feature enabled in DAPLink. This is done by writing the auto_on.cfg file to the USB mass storage 'MUSCA_B' while pressing nSRST button. See DAPLink docs above for more information.

Turn auto power on

There is a hidden command in the v3.4 firmware: Auto power can be turned on by writing auto_pwr.cfg to the USB mass storage 'MUSCA_B' while pressing nSRST button. Turning auto power off can be done by writing hard_pwr.cfg to the USB mass storage 'MUSCA_B' while pressing the nSRST button.




An overview of this board, as well as a Programming and Debugging guide can be found on the zyphyrproject documentation page.

NXP LPC55s69 Platform



The NXP Getting Started Guide can be used to build and run software.

Cypress Platform CY8CKIT-064S0S2-4343W



The board specification can be found here.

For Programming and Debugging, the onboard KitProg3 can be used. It is a programmer/debugger with USB-UART and USB-I2C funcionality. For details on the KitProg3 functionality see the KitProg3 User Guide.


To Program/Debug connect the board to thh PC using the USB cable. It will enumerate as a USB Composite Device if you are connecting to your PC for the first time.

KitProg3 operates in either CMSIS-DAP Bulk mode, or DAPLink mode (default). In SMSIS-DAP Bulk mode, two UART interfaces are supported. The status LED (Yellow) is always ON in CMSIS-DAP Bulk mode and ramping at a rate of 2 Hz in DAPLink mode. Press and release the Mode select button (SW3) to switch between these modes.

See the KitProg3 User Guide for more troubleshooting instructions.

Asurada Chromebook



The Asurada Chromebook should work out of the box; however there are Chromium OS docs for devoloper mode, as well as Closed Case Debug docs.

If you wish to flash a debug image you can use the following steps.

For CCD UART and flashing access, download the contents of this repository which contains scripts that automatically download and build required pieces.

  1. If these are installed on your machine, connect to the DUT via the SusyQable
    1. Start the servo daemon via servod -b asurada. Ensure that the SusyQable is in the correct orientation described in the trouble shooting section.
    2. Once the daemon has started leave it running in the background at all times for other steps.
  2. Next you may need to "unlock" the CCD on the device, which is a one-time manual step to be done on each device.
    1. Connect to the UART of the "Cr50" chip in the device
    2. Run dut-control cr50_uart_pty which should display the PTY number (such as cr50_uart_pty:/dev/pts/17)
    3. Use your terminal emulator of choice to connect to the PTY (e.g. minicom -w -D /dev/pts/17)
    4. Once connected, type ccd open
    5. The device may ask you to press the power button several times in a 5 minute window, whenever it says "press PP button now!" quickly press the power button.
    6. When the process is completed enter ccd reset factory
  3. With Servod running and CCD unlocked connect to the main SoC Uart
    1. Use dut-control cpu_uart_pty to find out the PTY number
  4. Use flashrom -n -w <image file> -p raiden_debug_spi:target=AP to flash an image file to the device
  5. To replace an image with a different build you need the cbfstool utility
    1. navigate to util/cbfstool, and run make
    2. To replace the image remove the existing bl31.elf file
    3. cbfstool <image file> remove -n fallback/bl31
    4. Re-add a different version
    5. cbfstool <image file> add-payload -n fallback/bl31 -f <new bl31.elf file>


When connecting to the Asurada Chromebook via the SusyQable, the DBG symbol should be in a upward position in the left USB-C port.

For more Chromebook devices you can look here. Currently the Asurada Chromebook is not on this list.

Lazor Chromebook



The Lazor Chromebook should work out of the box; however there are Chromium OS docs for devoloper mode, as well as Closed Case Debug docs.

If you wish to flash a debug image you can use the following steps.

For CCD UART and flashing access, download the contents of this repository which contains scripts that automatically download and build required pieces.

  1. If these are installed on your machine, connect to the DUT via the SusyQable
    1. Start the servo daemon via servod -b trogdor. Ensure that the SusyQable is in the correct orientation described in the trouble shooting section.
    2. Once the daemon has started leave it running in the background at all times for other steps.
  2. Next you may need to "unlock" the CCD on the device, which is a one-time manual step to be done on each device.
    1. Connect to the UART of the "Cr50" chip in the device
    2. Run dut-control cr50_uart_pty which should display the PTY number (such as cr50_uart_pty:/dev/pts/17)
    3. Use your terminal emulator of choice to connect to the PTY (e.g. minicom -w -D /dev/pts/17)
    4. Once connected, type ccd open
    5. The device may ask you to press the power button several times in a 5 minute window, whenever it says "press PP button now!" quickly press the power button.
    6. When the process is completed enter ccd reset factory
  3. With Servod running and CCD unlocked connect to the main SoC Uart
    1. Use dut-control cpu_uart_pty to find out the PTY number
  4. Use flashrom -n -w <image file> -p raiden_debug_spi:target=AP to flash an image file to the device
  5. To replace an image with a different build you need the cbfstool utility
    1. navigate to util/cbfstool, and run make
    2. To replace the image remove the existing bl31.elf file
    3. cbfstool <image file> remove -n fallback/bl31
    4. Re-add a different version
    5. cbfstool <image file> add-payload -n fallback/bl31 -f <new bl31.elf file>


When connecting to the Lazor Chromebook via the SusyQable, the DBG symbol should be in a downward position in the left USB-C port.

More board info can be found here.

Adding Boards to LAVA

"Adding a board to LAVA" can mean more than one thing, for example:

Once your device type is supported in LAVA, and the needed LAVA version is deployed to the Lab, you are ready to request that physical boards be installed in the Lab.

Hardware Requirements

The Lab has some basic Hardware Requirements for boards being installed in the lab. It's advisable to read the "Automation and hardware design" and "LAB Device Deployment Guide" pages for more detailed information:

Basic requirements:

How to get your board installed in the Linaro Cambridge Lab

Once your board is supported in the LAVA software, and your board meets the Hardware Requirements, you can raise an LSS ticket to get your board installed in the Lab.

  1. Go to!default.jspa
  2. Fill in the drop down boxes:
  3. Project: LAB & System Software (LSS)
  4. Issue Type: Ticket
  5. Click Next
  6. Fill in the required details
  7. Summary: You should fill in the "Summary" with a descriptive title. It may be a good idea to prefix titles with "TF CI: " to help identify them in the list of issues.
  8. Components: "LAB"
  9. Client Stakeholder: "Trusted-Firmware"
  10. Validation Server: ""
  11. Labels: "TrustedFirmware"
  12. Fill in the Description
  13. You will need to fill in the Description, even if you think the title is sufficient. Provide enough overview detail so the request is clear to understand by management, but make sure you include all the technical details you need for the support engineer to install your board.
  14. If you think you will need specific hardware, such as a dedicated dispatcher, an SDmux, etc. then please describe that here.
  15. Specify the type and number of boards you wish to be installed.
  16. Click the "Create" button at the bottom of the page
  17. Add Watchers
  18. It's probably a good idea to add Don Harbin to the Watchers on the ticket.


SQUAD is a frontend to LAVA, featuring a proxy for job submission, a database storing test results from LAVA jobs, and a dashboard where results can be compared across CI jobs and metrics generated.

The top level SQUAD project for TrustedFirmware is here:


TF-A has several SQUAD projects, the most interesting is tf-main:

There are other project, but the details


Staging Trusted Firmware System

This was documented here, but has been copied here as we approach wider review:

Brief description of the setup

This document does not go into detail about each project, and is meant to be used as guidelines and rules for accessing the next environment.


The staging setup or “next” is meant to be used for developers to be able to test the CI infrastructure. The setup has been set up exactly the same as the production environment, the major difference between them is developers use a staging Jenkins server instead of the production server.

Rules & Environment setup

Staging environments have been set up in the next/* namespace location:

The next/* namespace is mirrored from production. The only repository that is not mirrored is the tf-*-job-configs repos. All other repositories are mirrored and as such the user should branch out from master.

Users need to be placed in the trusted-firmware-staging-approvers Gerrit group. This is done by creating a ticket, please see instructions at the bottom of the Open CI wiki page. This group allows users to have submit and merge (+2) writes to all repositories under the next/* namespace, without needing any approval from a peer.

Due to the nature of allowing users to self approve their submit/merge changes into Gerrit, it is important that users understand that it triggers Jenkins jobs and as such care has to be taken when deploying those changes.

Basic rules all developers should follow:

How to setup basic next environment

You have two options:

It might be easier to just add a /next/ remote to the user's existing repo clone and work from that. However, the /next/ has had the “basic rules” applied, and as such it is important that the user does not break these rules.

tf-m-job-configs and tf-a-job-configs are not mirrored from production. However the other repos are, and as such the user can branch out from master and develop from there.

To add a remote, it is simply necessary to add /next/ the url. So

ssh:// becomes

ssh:// This then can be added with ‘git remote add gerrit-next

ssh://’ or cloned via git clone &lturl>.

Once a remote has been added, the user can then do git fetch gerrit-next and then checkout to that branch.

Sample script to clone the repositories:


set -e

for project in tf-a-ci-scripts tf-a-job-configs tf-m-ci-scripts tf-m-job-configs; do
  git clone "ssh://${username}${project}"
  cd ${project} 
  git remote add gerrit-next ssh://${username}${project}
  git fetch gerrit-next
  cd ..

It's recommended that the user read for understanding two remotes.

Once in this environment, it is recommended the user then checkouts a new dev location and works from that. then copy the \.yaml file. This should be the same for the \ location too. Once set up it is recommended that the user appends these changes to the job config.

  - authorization:
          - job-read
          - job-extended-read
       - job-read
       - job-extended-read
       - job-build
       - job-cancel

It is important to note here, the user needs to replace bhcopeland with your own GitHub username. From this, it allows you to manually trigger and canel the job.

Please ensure any triggers (timed based etc) are disabled. And please ensure silent: true is set inside the gerrit trigger so no gerrit comments get triggered.

Workflow for next/tf-a-job-configs.git

This is the workflow for creating ‘per-user’ jenkins jobs in staging instance. next/ci/tf-a-job-configs.git repository should be used in this case.

Alt text

Workflow for other repositories

Other repositories, that are used inside the jobs, can be copied to other server (for example This should be added as a new remote to the existing repository. After changes are made and work well, they should be sent for review. Example below:

Alt text

Similar workflow should be used when migrating changes to ci/tf-a-scripts and ci/tf-a-job-configs repositories. Changes in the next/* should be sent for review against repositories in the ci/* path.

As noted above, changes in next/tf-a-job-configs can be self approved and merged. Changes in user repositories can be pushed without reviews.

Uploading files into shared repository

For those files that are prerequisites for CI execution, i.e. SCP RAM, Code Coverage FVP plugin, etc. are shared through and tuxput is the service which allows users to directly upload to S3 without having AWS credentials into the latter site.

Tuxput Installation

The tpcli command is distributed as part of the tuxput python package. To install, run:

$ pip install tuxput

Upload A File


$ tpcli -t <token> -b trustedfirmware-prod-storage file

where tuxput instance's endpoint is at*

Anything specified after the upload/ is interpreted as a folder to use for the file that has been uploaded. If you have permission to write to that path and the folder doesn't exist then it is created. If the upload target is a directory instead of a file, tpcli recurses through the directory and add it to the folder prefix. When tuxput has finished attempting to upload, it prints out a list of the results for each file. Successful uploads reports as HTTP 204, and if it was unsuccessful the error message will be displayed.


  1. Single file upload:
$ touch sample.txt
$ tpcli -t <token> -b trustedfirmware-prod-storage sample.txt

  1. Directory upload:
$ mkdir demo2
$ mkdir demo2/sub
$ touch demo2/sample1.txt
$ touch demo2/sub/sample2.txt
$ tpcli -t <token> -b trustedfirmware-prod-storage demo2

  1. Directory upload with a new folder in the path:
$ mkdir demo2
$ mkdir demo2/sub
$ touch demo2/sample1.txt
$ touch demo2/sub/sample2.txt
$ tpcli -t <token> -b trustedfirmware-prod-storage demo2


Obtaining a token

In order to obtain a token, please open an LSS ticket.

Overwrite a file

The ability to overwrite previous uploads has been enabled on this instance of tuxput.

Delete a file

Tuxput does not currently offer a way to delete a file. If you need a file or folder deleted, please submit an LSS ticket so that we can manually remove it.

Future Updates

The project is being hosted at .If you run into a bug or have a feature request, please submit it there.

FVP Docker Images

LAVA is used to launch FVP (Fast Models) based on CI artifacts. In turn, models are launched inside a containerized environment. In other words, models are not run from a bare metal host system, but inside containers on a LAVA dispatcher, where these are launched by LAVA using CI artifacts and configured model parameters. In case a new FVP Docker Image is required at the CI, a contributor needs to download the corresponding FVP model tarball, the installer, and upload it to a CI private repository, then a CI job would automatically generate the docker image.

FVP Models

FVP model are commonly available at these sites:

Download the required model, e.g. FVP_Base_RevC-2xAEMvA_11.15_14.tgz. Once download is completed, install tuxput package and run the following command using your personal token

tpcli -t <token> -b trustedfirmware-fvp <fvp model>

for the case of the FVP_Base_RevC-2xAEMvA_11.15_14.tgz, command becomes

tpcli -t <token> -b trustedfirmware-fvp FVP_Base_RevC-2xAEMvA_11.15_14.tgz

If no errors are seen, the corresponding CI job would detect this new model in the repository and automatically creates the corresponding docker image. This is a cron job executed in a daily basis but can be also triggered manually: go to the CI job landing page and just click the 'Build now' icon.

Misc Info

LAVA Ready

This section documents the high-level guidelines and expectations when a TrustedFirmware Member wishes to put a board into the OpenCI board farm. This board farm is located in the Linaro facility near Cambridge, UK. A section of the Linaro lab has been allocated to All trustedfirmware boards are collocated on racks dedicated to the project.

The general steps/guidelines for getting hardware placed into the lab is as follows: