[edk2-devel] EDK2 Host-Based Unit Test RFC (Now with docs!)

Fri Nov 22 07:39:23 UTC 2019

Now that CI has landed in edk2/master, we'd like to get the basic framework for unittesting staged as well. Target intercept date would be immediately after the 2019/11 stabilization, so we wanted to go ahead and get comments now.

The host unittest framework consists of five primary pieces:
- The test library (Cmocka)
- Support libraries (Found in UnitTestPkg)
- The test support plugins (HostUnitTestComilerPlugin, HostUnitTestDxeCompleteCheck, HostBasedUnitTestRunner)
- The configuration in the package-based *.ci.yaml file and package-based Test.dsc
- The tests themselves

We have a demo branch set up at:
https://github.com/corthon/edk2-staging/tree/edk2-host-test_v2
We also have a demo build pipeline at:
https://dev.azure.com/tianocore/edk2-ci-play/_build?definitionId=36&_a=summary<https://nam06.safelinks.protection.outlook.com/?url=https%3A%2F%2Fdev.azure.com%2Ftianocore%2Fedk2-ci-play%2F_build%3FdefinitionId%3D36%26_a%3Dsummary&data=02%7C01%7CBret.Barkelew%40microsoft.com%7C3ce0b4eaf6d14de8822808d769f2c5fd%7C72f988bf86f141af91ab2d7cd011db47%7C1%7C0%7C637094365265670742&sdata=HFBmk%2FdB5pXI3exxB82pTS1oF877fLsrrdcirOzCCw0%3D&reserved=0>

The current demo branch contains a single test in MdePkg for SafeIntLib (module file MdePkg\Test\UnitTest\Library\BaseSafeIntLib\TestBaseSafeIntLib.inf). This test is automatically detected by the HostUnitTestComilerPlugin and run by the HostBasedUnitTestRunner as part of the CI process.

A few notes about the current demo:
1) The demo currently only works on Windows build chains, but there's no reason to believe that it can't work equally well on Linux build chains, and are happy to work with anyone to get it there.

2) The demo currently has four failing conditions that can be seen towards the end of MdePkg "Build and Test" log file for this build:
https://dev.azure.com/tianocore/edk2-ci-play/_build/results?buildId=2813<https://nam06.safelinks.protection.outlook.com/?url=https%3A%2F%2Fdev.azure.com%2Ftianocore%2Fedk2-ci-play%2F_build%2Fresults%3FbuildId%3D2590&data=02%7C01%7CBret.Barkelew%40microsoft.com%7C3ce0b4eaf6d14de8822808d769f2c5fd%7C72f988bf86f141af91ab2d7cd011db47%7C1%7C0%7C637094365265670742&sdata=HgPEoYg%2Fpx0fv4J5ULO1p0kvLfqcySkJHAYxp9GB598%3D&reserved=0>
"WARNING -   Test SafeInt16ToChar8 - Status
d:\a\1\s\MdePkg\Test\UnitTest\Library\BaseSafeIntLib\TestBaseSafeIntLib.c:302: error: Failure!
WARNING - TestBaseSafeIntLib.exe Test Failed
WARNING -   Test SafeInt32ToChar8 - Status
d:\a\1\s\MdePkg\Test\UnitTest\Library\BaseSafeIntLib\TestBaseSafeIntLib.c:638: error: Failure!
WARNING - TestBaseSafeIntLib.exe Test Failed
WARNING -   Test SafeIntnToChar8 - Status
d:\a\1\s\MdePkg\Test\UnitTest\Library\BaseSafeIntLib\TestBaseSafeIntLib.c:1051: error: Failure!
WARNING - TestBaseSafeIntLib.exe Test Failed
WARNING -   Test SafeInt64ToChar8 - Status
d:\a\1\s\MdePkg\Test\UnitTest\Library\BaseSafeIntLib\TestBaseSafeIntLib.c:1456: error: Failure!"
These failures seem to be legitimate and further work should be done by the community to decide whether the test case is correct or the library is correct, but one of them needs to change.

3) Current demo pulls in test collateral from a fork of the edk2-test repo. This repo is currently *very* heavy due to the history of the UEFI SCT project and the number of binaries that it pulls down. It's possible that we (the community) need to select a better place for this code to live. Maybe in edk2 primary (though it's not explicitly firmware code, so it seems unnecessary). Maybe in a new edk2-test2 repo or something like that.

There’s an RFC doc here: https://github.com/corthon/edk2-staging/blob/edk2-host-test_v2/Readme-RFC.md
And a usage guide here: https://github.com/corthon/edk2-staging/blob/edk2-host-test_v2/UnitTestPkg/ReadMe.md

Once again, would love to get this into EDK2 master after stabilization, and most of this has previously been shopped around in other discussion threads. This is just where the rubber meets the road and is the minimal subset of code that needs to land for folks to start contributing unittests against the core libraries that can be run as part of any CI pass.

Thanks!
- Bret

-=-=-=-=-=-=-=-=-=-=-=-
Groups.io Links: You receive all messages sent to this group.

View/Reply Online (#51165): https://edk2.groups.io/g/devel/message/51165
Mute This Topic: https://groups.io/mt/61559421/1813853
Group Owner: devel+owner at edk2.groups.io
Unsubscribe: https://edk2.groups.io/g/devel/unsub  [edk2-devel-archive at redhat.com]
-=-=-=-=-=-=-=-=-=-=-=-

-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://listman.redhat.com/archives/edk2-devel-archive/attachments/20191122/9621e290/attachment.htm>