Slash Unit Testing Time in Embedded Software Regression Testing

Accurate and efficient tests are essential when developing software for embedded systems, particularly in continuous integration and functional safety flows. Quick pass/fail results from a comprehensive regression test suite enable teams to develop quality software without wasting engineering resources waiting for tests to finish running. Unit tests are a key component of regression test suites as they ensure the small code snippets, the building blocks of the product, are working as expect.

Unit testing is important to streamline as it can become a speed bottleneck with hundreds or even thousands of individual tests running to verify each code snippet is working as expected. When running a regression test suite becomes too time-consuming, engineering resources can be unused waiting for tests to finish, code quality can decrease with code tested less frequently, and time-to-market is increased resulting in less profits.

This blog will detail a testing methodology that runs faster and more efficiently than traditional approaches by leveraging virtual platforms.

Why virtual platforms?

Virtual platforms offer multiple values over hardware for the types of tests that are intended to run quickly on small sets of code, such as unit or integration tests. These are:

  • Speed - virtual platforms have no overhead for flashing the application on physical hardware, leading to significant time saved for small and fast unit tests.
  • Scalability - virtual platforms can scale to a running a large number of tests in parallel more cost-effectively than purchasing a farm of physical hardware.
  • Maintainability - virtual platforms never overheat, wear out from overuse, break from misuse, or take up physical space and resources; all downsides faced by physical hardware.
  • Upgradability - virtual platforms can easily be changed and reconfigured to match corresponding changes to the underlying hardware platform under development; these changes can be costly or impossible with physical hardware.

That isn’t to say hardware has no place in the continuous integration regression test suite. Leveraging virtual platforms for unit and integration testing while running other integration and system tests on hardware can lead to a very efficient software development workflow. Testing on virtual platforms is fast and accurate but may not contain specific components necessary for full system tests. Because there is a typically a lower number of full system tests than small unit tests, running system tests on hardware gives the benefits of a complete system performance check with fewer physical boards in the regression test suite. 

To compare how running unit tests on virtual platforms results in higher performance than running on hardware, let’s use the same example regression test suite for both platforms and analyze the results. The open-source embedded operating system Mbed OS provides with it a suite of tests to run. These validate that various parts of the platform in question--hardware or virtual--are working properly with the various Mbed OS capabilities.

Mbed OS is intended for Cortex-M micro-controllers and supports a wide range of Arm Cortex-M based devices, including the MPS2+ FPGA board from Arm. This FPGA can be programmed to use various Arm Cortex-M cores. Mbed OS also supports easily running tests on virtual platform representing this MPS2+ FPGA board, known as a Fixed Virtual Platform (FVP). Because unit tests often rely on only the core and memory, platforms with different peripherals can still be used to test the same code--another reason why virtual platforms are so powerful.

In this example we will test using an Arch Pro board and a MPS2+ FVP, both containing a Cortex-M3 core. The subset of unit tests used now require only the core and memory, so the MPS2+ FVP is suitable to run the same types of tests intended for the Arch Pro board. By obtaining the free Mbed OS tools and a temporary license to Arm Compiler and the MPS2+ FVP you can compare testing results between hardware and a virtual platform right away.

Required tools and software

Pre-requisites

The following software and hardware is assumed to already be installed:

  • Python 2.7.12 or later (not Python 3!).
  • An Arm compiler (Arm Compiler 5 used in this example and should be in your PATH. Arm Compiler 6 and GCC also work).
  • A Mbed OS compatible board to test with. This example will use the Arch Pro board from Seeed Studio.

Virtual Platforms and Arm Compiler

There are a few ways to get access to the MPS2+ FVPs. Using the FVPs that come with Arm Development Studio has the added benefit of including Arm Compiler 5 and Arm Compiler 6, so that is the path of least resistance for this particular example. See this guide for obtaining a license and downloading the tool, which will set the correct compiler path and pre-built FVPs. Note the path to these FVPs, as Mbed OS will need a pointer to its location to use it during testing. They will be under '<Arm_Development_Studio_path>/bin/'.

Mbed OS Components

Create a new directory somewhere--note on Windows not to make the path name too long, as the limit on Windows 10 is 260 characters by default for any path. Inside that directory run the following commands to install various Mbed OS components:

Mbed OS 

This is the main Mbed OS library, containing source code and test files. Verify the version is at or above 1.8.2. 

git clone https://github.com/ARMmbed/mbed-os
mbed --version
 

Greentea

This is a Mbed OS test harness. Verify the version is at or above 1.4.0 and that the --fm switch is present.

git clone https://github.com/ARMmbed/greentea		
cd greentea
sudo python setup.py install
cd ..
mbedgt --version
mbedgt --fm

Mbed CLI

This is the Arm Mbed command-line tool.

Linux and Mac users

sudo pip install mbed-cli

Windows users

Download and run executable

mbed-fastmodel-agent

This is the add-on that enables Arm virtual platforms as targets, specifically the MPS2+ FVP.

git clone https://github.com/ARMmbed/mbed-fastmodel-agent

After cloning from the repository, edit the file “mbed-fastmodel-agent\fm_agent\settings.json” to point to the folder containing the FVPs. In the file, change the value assigned to “model_lib_path” for your OS. Once this is completed, run the setup script and check that the FVP with the relevant core is listed as available (the FVP_MPS2_M3 in this example) by running 'mbedfm'.

cd mbed-fastmodel-agent
python setup.py install
cd ..
mbedfm

Building and running the unit tests

With the proper tools installed we can begin building and executing the provided tests. First, we compile the tests for both the Arch Pro hardware and MPS2+ FVP. These steps will take some time as Mbed OS compiles the entire library to run the unit tests properly on target. 

For the Arch Pro, plug in the board over USB and run the following commands:

mbedls
cd mbed-os
mbed test -t ARM -m ARCH_PRO --compile

mbedls will verify that you have a mbed enabled board connected to your computer. The mbed test command will compile all tests with the Arm Compiler.

For the MPS2+ FVP, run the following commands:

cd mbed-os
mbed test -t ARM -m FVP_MPS2_M3 --compile

Now we are ready to run the unit test suite on both hardware and virtual platforms to contrast the results. Important to note is that certain tests will run on one platform and not another due to differences in the internal hardware. For example, the Arch Pro cannot run some real-time operating system tests and driver tests that it does not support with its hardware. To ensure the Arch Pro and MPS2+ FVP are running the same test suite the “-n” switch is added to enumerate the tests to run, with wildcards (*) used to include all of a certain class of test and commas to separate the tests. You must be in the 'mbed-os' directory to perform these tests.

To run on the Arch Pro:

mbedgt -m ARCH_PRO -n features-device_key*,features-frameworks*,features-storage*,tests-integration*,tests-mbed_platform*

To run on the MPS2+ FVP:

mbedgt --fm FVP_MPS2_M3:DEFAULT -n features-device_key*,features-frameworks*,features-storage*,tests-integration*,tests-mbed_platform*

All test suites should pass, and a total time will be displayed after completing all tests. This total test time for both the Arch Pro board and the MPS2+ FVP will vary slightly based on the host computer bandwidth and performance. The test commands above run 33 test suites with a total of 124 tests on each platform. Here are the times to complete all tests for each platform on my machine – you may get different depending on your machine performance.

  • Arch Pro: 488 seconds
  • MPS2+ FVP: 259 seconds

These runs test the same functionality, and because Arm virtual platforms are 100% functionally accurate the total test time is cut almost in half while losing no information. The speed increase when using virtual platforms compared to hardware can be attributed to a number of factors, including no flash time and a faster clock speed on virtual platforms running on a host OS than a development board.

To speed up regression tests even more, running virtual platforms in parallel will reduce the testing time dramatically. This is another substantial benefit of using virtual platforms for running tests--scaling with more simulations in parallel to speed up/run more tests is simple and easily maintained as opposed to buying and managing more hardware boards. Running simulations in parallel with Mbed OS is straightforward, adding the --parallel switch with the number of runs to perform in parallel. To run 2 tests in parallel on the previous test suite, run the following command:

mbedgt --fm FVP_MPS2_M3:DEFAULT -n features-device_key*,features-frameworks*,features-storage*,tests-integration*,tests-mbed_platform* --parallel 2

Here are the results of the Arch Pro, MPS2+ FVP, and MPS2+ FVP running 2 tests in parallel:

Platform Time (seconds) % Speed Increase from Hardware
Arch Pro Hardware  488 0%
MPS2+ FVP 259 88%
MPS2+ FVP (2 in parallel) 160 205%

The time a regression test suite takes to run decreases by half for each simulation running in parallel, limited by the computation power of the host machine and the size of the test suite. Running 3 tests in parallel will result in approximately a 300% speed-up, 4 tests in parallel a ~400% speed-up, etc. Virtual platforms can bring this same regression suite execution speed increase to most any embedded software development testing infrastructure without sacrificing functional accuracy.

For these unit tests only consisting of the core and memory interaction, using a Fixed Virtual Platform with the same core but different peripherals than the desired hardware being developed is a valid development methodology. In cases where peripherals must be used or tested, however, the need to customize these virtual platforms becomes essential. Arm Fast Models, the underlying technology comprising Arm FVPs, allows customization of system designs to better match your target systems. For full flexibility, the Arm Fast Models library comes with a multitude of Arm components already modeled and the option to model custom components manually in SystemC. For a quick-start guide on creating your own virtual platforms with Arm Fast Models to better represent specific hardware peripherals in your tests, see this guide for obtaining a license and getting starting immediately:

Arm Fast Models Get Started Guide

Summary

Virtual platforms from Arm enable regression suites to be run faster in a more scalable and maintainable manner compared to hardware. As Arm Fast Models contained in FVPs are validated against Arm RTL, using Arm-based virtual platforms results in a 100% functionally accurate simulation. Leveraging virtual platforms in parallel as part of a robust regression test suite will reduce product time-to-market while increasing code quality.

Anonymous