Accurate and efficient tests are essential when developing software for embedded systems, particularly in continuous integration and functional safety flows. Quick pass/fail results from a comprehensive regression test suite enable teams to develop quality software without wasting engineering resources waiting for tests to finish running. Unit tests are a key component of regression test suites as they ensure the small code snippets, the building blocks of the product, are working as expect.
Unit testing is important to streamline as it can become a speed bottleneck with hundreds or even thousands of individual tests running to verify each code snippet is working as expected. When running a regression test suite becomes too time-consuming, engineering resources can be unused waiting for tests to finish, code quality can decrease with code tested less frequently, and time-to-market is increased resulting in less profits.
This blog will detail a testing methodology that runs faster and more efficiently than traditional approaches by leveraging virtual platforms.
Virtual platforms offer multiple values over hardware for the types of tests that are intended to run quickly on small sets of code, such as unit or integration tests. These are:
That isn’t to say hardware has no place in the continuous integration regression test suite. Leveraging virtual platforms for unit and integration testing while running other integration and system tests on hardware can lead to a very efficient software development workflow. Testing on virtual platforms is fast and accurate but may not contain specific components necessary for full system tests. Because there is a typically a lower number of full system tests than small unit tests, running system tests on hardware gives the benefits of a complete system performance check with fewer physical boards in the regression test suite.
To compare how running unit tests on virtual platforms results in higher performance than running on hardware, let’s use the same example regression test suite for both platforms and analyze the results. The open-source embedded operating system Mbed OS provides with it a suite of tests to run. These validate that various parts of the platform in question--hardware or virtual--are working properly with the various Mbed OS capabilities.
Mbed OS is intended for Cortex-M micro-controllers and supports a wide range of Arm Cortex-M based devices, including the MPS2+ FPGA board from Arm. This FPGA can be programmed to use various Arm Cortex-M cores. Mbed OS also supports easily running tests on virtual platform representing this MPS2+ FPGA board, known as a Fixed Virtual Platform (FVP). Because unit tests often rely on only the core and memory, platforms with different peripherals can still be used to test the same code--another reason why virtual platforms are so powerful.
In this example we will test using an Arch Pro board and a MPS2+ FVP, both containing a Cortex-M3 core. The subset of unit tests used now require only the core and memory, so the MPS2+ FVP is suitable to run the same types of tests intended for the Arch Pro board. By obtaining the free Mbed OS tools and a temporary license to Arm Compiler and the MPS2+ FVP you can compare testing results between hardware and a virtual platform right away.
The following software and hardware is assumed to already be installed:
There are a few ways to get access to the MPS2+ FVPs. Using the FVPs that come with Arm Development Studio has the added benefit of including Arm Compiler 5 and Arm Compiler 6, so that is the path of least resistance for this particular example. See this guide for obtaining a license and downloading the tool, which will set the correct compiler path and pre-built FVPs. Note the path to these FVPs, as Mbed OS will need a pointer to its location to use it during testing. They will be under '<Arm_Development_Studio_path>/bin/'.
Create a new directory somewhere--note on Windows not to make the path name too long, as the limit on Windows 10 is 260 characters by default for any path. Inside that directory run the following commands to install various Mbed OS components:
This is the main Mbed OS library, containing source code and test files. Verify the version is at or above 1.8.2.
git clone https://github.com/ARMmbed/mbed-os mbed --version
This is a Mbed OS test harness. Verify the version is at or above 1.4.0 and that the --fm switch is present.
git clone https://github.com/ARMmbed/greentea cd greentea sudo python setup.py install cd .. mbedgt --version mbedgt --fm
This is the Arm Mbed command-line tool.
Linux and Mac users
sudo pip install mbed-cli
Windows users
> Download and run executable
This is the add-on that enables Arm virtual platforms as targets, specifically the MPS2+ FVP.
git clone https://github.com/ARMmbed/mbed-fastmodel-agent
After cloning from the repository, edit the file “mbed-fastmodel-agent\fm_agent\settings.json” to point to the folder containing the FVPs. In the file, change the value assigned to “model_lib_path” for your OS. Once this is completed, run the setup script and check that the FVP with the relevant core is listed as available (the FVP_MPS2_M3 in this example) by running 'mbedfm'.
cd mbed-fastmodel-agent python setup.py install cd .. mbedfm
With the proper tools installed we can begin building and executing the provided tests. First, we compile the tests for both the Arch Pro hardware and MPS2+ FVP. These steps will take some time as Mbed OS compiles the entire library to run the unit tests properly on target.
For the Arch Pro, plug in the board over USB and run the following commands:
mbedls cd mbed-os mbed test -t ARM -m ARCH_PRO --compile
mbedls will verify that you have a mbed enabled board connected to your computer. The mbed test command will compile all tests with the Arm Compiler.
For the MPS2+ FVP, run the following commands:
cd mbed-os mbed test -t ARM -m FVP_MPS2_M3 --compile
Now we are ready to run the unit test suite on both hardware and virtual platforms to contrast the results. Important to note is that certain tests will run on one platform and not another due to differences in the internal hardware. For example, the Arch Pro cannot run some real-time operating system tests and driver tests that it does not support with its hardware. To ensure the Arch Pro and MPS2+ FVP are running the same test suite the “-n” switch is added to enumerate the tests to run, with wildcards (*) used to include all of a certain class of test and commas to separate the tests. You must be in the 'mbed-os' directory to perform these tests.
To run on the Arch Pro:
mbedgt -m ARCH_PRO -n features-device_key*,features-frameworks*,features-storage*,tests-integration*,tests-mbed_platform*
To run on the MPS2+ FVP:
mbedgt --fm FVP_MPS2_M3:DEFAULT -n features-device_key*,features-frameworks*,features-storage*,tests-integration*,tests-mbed_platform*
All test suites should pass, and a total time will be displayed after completing all tests. This total test time for both the Arch Pro board and the MPS2+ FVP will vary slightly based on the host computer bandwidth and performance. The test commands above run 33 test suites with a total of 124 tests on each platform. Here are the times to complete all tests for each platform on my machine – you may get different depending on your machine performance.
These runs test the same functionality, and because Arm virtual platforms are 100% functionally accurate the total test time is cut almost in half while losing no information. The speed increase when using virtual platforms compared to hardware can be attributed to a number of factors, including no flash time and a faster clock speed on virtual platforms running on a host OS than a development board.
To speed up regression tests even more, running virtual platforms in parallel will reduce the testing time dramatically. This is another substantial benefit of using virtual platforms for running tests--scaling with more simulations in parallel to speed up/run more tests is simple and easily maintained as opposed to buying and managing more hardware boards. Running simulations in parallel with Mbed OS is straightforward, adding the --parallel switch with the number of runs to perform in parallel. To run 2 tests in parallel on the previous test suite, run the following command:
mbedgt --fm FVP_MPS2_M3:DEFAULT -n features-device_key*,features-frameworks*,features-storage*,tests-integration*,tests-mbed_platform* --parallel 2
Here are the results of the Arch Pro, MPS2+ FVP, and MPS2+ FVP running 2 tests in parallel:
The time a regression test suite takes to run decreases by half for each simulation running in parallel, limited by the computation power of the host machine and the size of the test suite. Running 3 tests in parallel will result in approximately a 300% speed-up, 4 tests in parallel a ~400% speed-up, etc. Virtual platforms can bring this same regression suite execution speed increase to most any embedded software development testing infrastructure without sacrificing functional accuracy.
For these unit tests only consisting of the core and memory interaction, using a Fixed Virtual Platform with the same core but different peripherals than the desired hardware being developed is a valid development methodology. In cases where peripherals must be used or tested, however, the need to customize these virtual platforms becomes essential. Arm Fast Models, the underlying technology comprising Arm FVPs, allows customization of system designs to better match your target systems. For full flexibility, the Arm Fast Models library comes with a multitude of Arm components already modeled and the option to model custom components manually in SystemC. For a quick-start guide on creating your own virtual platforms with Arm Fast Models to better represent specific hardware peripherals in your tests, see this guide for obtaining a license and getting starting immediately:
Arm Fast Models Get Started Guide
Virtual platforms from Arm enable regression suites to be run faster in a more scalable and maintainable manner compared to hardware. As Arm Fast Models contained in FVPs are validated against Arm RTL, using Arm-based virtual platforms results in a 100% functionally accurate simulation. Leveraging virtual platforms in parallel as part of a robust regression test suite will reduce product time-to-market while increasing code quality.
Thanks for getting back so quickly. That's quite a lot of guesswork that I should have made, is there anyway to make it more clear what a license of Fast Models is required (which does not seem to be the same as a license for Arm Development Studio) and a few more steps are required ? Else, someone reaching this blog post thinking he or she will save some time on testing, will just end up spending three days just to make existing tests work (without success so far).Regarding the mbed-fastmodel-agent tool, is there any plan to use the Network Model instead of Library Model in the future ? Or at least being able to choose ? It may not be as effective but for evaluation purpose it would be easier as the setup process would be quite simple.
Laurent Louf
Thanks for giving it a try! I hope you find as much use of this workflow as the MbedOS team does.
For your question, the current iteration of mbed-fastmodel-agent needs the model libs (.so or .dll files), not the executables of the models. This was done for performance & scalability reasons. That being said, you will have to generate these .so or .dll files yourself via our Fast Models tool. Using an evaluation license to Fast Models you can open up the MPS2+ FVPs and generate the lib files. Here is how to do that:
- Get the latest FM Evaluation Package from https://developer.arm.com/tools-and-software/simulation-models/fast-models
- Ensure you have Visual Studio 2015 to compile Fast Models
- Install Fast Models by following the install instructions
- Once installed, open the System Canvas tool (enables you to build systems) by typing 'sgcanvas' into your CMD line or terminal. Make sure to open it as an admin or sudo if you are in a write-controlled directory from the install otherwise you will have trouble saving changes or creating new files.
- Click 'open' in the top left and navigate to the first FVP you want to try (if it is Cortex-M7, navigate to the path similar to C:\Program Files\ARM\FastModelsPortfolio_11.5\examples\LISA\FVP_MPS2\Build_Cortex-M7 on your system) and select 'FVP_MPS2_Cortex-M7.sgproj'
- In the menu bar click 'Settings' > 'Targets' and check 'CADI library'. Make sure to uncheck any other boxes as the tool can only generate one target at a time. Click 'Apply' and 'OK'.
- Click 'Save All' in the menu bar
- Click 'Build'
This should build a .so file if on Linux and a .dll file if on Windows of the MPS2+ FVP for your desired core. It will be located in the build directory just created and called something like 'cadi_system_Win64-Release-VC2015.dll' depending on your OS and compiler.
A long but effective process. Please let me know if you have other questions!
Best,
-Zach
Hey Zach Lasiuk , I've tried to follow this tutorial, but I'm stuck at this step : "mbed-fastmodel-agent" . I've downloaded Arm Development Studio, installed it, got an evaluation license, but I can't find the model libs path to set in the configuration file. I've searched on my computer for any of the files, they are nowhere to be found. I have the executables corresponding to the different models, but not the libs. Any help ?