[edk2-devel] [PATCH v2 1/1] UnitTestFrameworkPkg: Add info to readme about working with UnitTests
Michael D Kinney
michael.d.kinney at intel.com
Fri May 29 18:36:13 UTC 2020
Reviewed-by: Michael D Kinney <michael.d.kinney at intel.com>
Mike
> -----Original Message-----
> From: Bret Barkelew <bret at corthon.com>
> Sent: Friday, May 29, 2020 10:34 AM
> To: devel at edk2.groups.io
> Cc: Kinney, Michael D <michael.d.kinney at intel.com>;
> Bret Barkelew <bret.barkelew at microsoft.com>
> Subject: [PATCH v2 1/1] UnitTestFrameworkPkg: Add info
> to readme about working with UnitTests
>
> Cc: Michael D Kinney <michael.d.kinney at intel.com>
>
> Cc: Bret Barkelew <bret.barkelew at microsoft.com>
>
> Signed-off-by: Bret Barkelew
> <bret.barkelew at microsoft.com>
>
> ---
>
>
>
> Notes:
>
> v2:
>
> - Update the SpellCheck YAML to incude GitHub
> handles in the dictionary
>
>
>
> .pytool/Readme.md |
> 125 +++++++++++++++-----
>
> UnitTestFrameworkPkg/ReadMe.md |
> 82 ++++++++++++-
>
> UnitTestFrameworkPkg/UnitTestFrameworkPkg.ci.yaml |
> 7 +-
>
> 3 files changed, 177 insertions(+), 37 deletions(-)
>
>
>
> diff --git a/.pytool/Readme.md b/.pytool/Readme.md
>
> index c7dce3b64ca0..c401dba18fbf 100644
>
> --- a/.pytool/Readme.md
>
> +++ b/.pytool/Readme.md
>
> @@ -2,31 +2,32 @@
>
>
>
> ## Basic Status
>
>
>
> -| Package | Windows VS2019 (IA32/X64)|
> Ubuntu GCC (IA32/X64/ARM/AARCH64) | Known Issues |
>
> -| :---- | :----- | :--
> -- | :--- |
>
> -| ArmPkg |
>
> -| ArmPlatformPkg |
>
> -| ArmVirtPkg | SEE PACKAGE README | SEE
> PACKAGE README |
>
> -| CryptoPkg | :heavy_check_mark: |
> :heavy_check_mark: | Spell checking in audit mode
>
> -| DynamicTablesPkg |
>
> -| EmbeddedPkg |
>
> -| EmulatorPkg | SEE PACKAGE README | SEE
> PACKAGE README | Spell checking in audit mode
>
> -| FatPkg | :heavy_check_mark: |
> :heavy_check_mark: |
>
> -| FmpDevicePkg | :heavy_check_mark: |
> :heavy_check_mark: |
>
> -| IntelFsp2Pkg |
>
> -| IntelFsp2WrapperPkg |
>
> -| MdeModulePkg | :heavy_check_mark: |
> :heavy_check_mark: | DxeIpl dependency on ArmPkg,
> Depends on StandaloneMmPkg, Spell checking in audit
> mode
>
> -| MdePkg | :heavy_check_mark: |
> :heavy_check_mark: | Spell checking in audit mode
>
> -| NetworkPkg | :heavy_check_mark: |
> :heavy_check_mark: | Spell checking in audit mode
>
> -| OvmfPkg | SEE PACKAGE README | SEE
> PACKAGE README | Spell checking in audit mode
>
> -| PcAtChipsetPkg | :heavy_check_mark: |
> :heavy_check_mark: |
>
> -| SecurityPkg | :heavy_check_mark: |
> :heavy_check_mark: | Spell checking in audit mode
>
> -| ShellPkg | :heavy_check_mark: |
> :heavy_check_mark: | Spell checking in audit mode, 3
> modules are not being built by DSC
>
> -| SignedCapsulePkg |
>
> -| SourceLevelDebugPkg |
>
> -| StandaloneMmPkg |
>
> -| UefiCpuPkg | :heavy_check_mark: |
> :heavy_check_mark: | Spell checking in audit mode, 2
> binary modules not being built by DSC
>
> -| UefiPayloadPkg |
>
> +| Package | Windows VS2019 (IA32/X64)|
> Ubuntu GCC (IA32/X64/ARM/AARCH64) | Known Issues |
>
> +| :---- | :----- | :-
> --- | :--- |
>
> +| ArmPkg |
>
> +| ArmPlatformPkg |
>
> +| ArmVirtPkg | SEE PACKAGE README | SEE
> PACKAGE README |
>
> +| CryptoPkg | :heavy_check_mark: |
> :heavy_check_mark: | Spell checking in audit mode
>
> +| DynamicTablesPkg |
>
> +| EmbeddedPkg |
>
> +| EmulatorPkg | SEE PACKAGE README | SEE
> PACKAGE README | Spell checking in audit mode
>
> +| FatPkg | :heavy_check_mark: |
> :heavy_check_mark: |
>
> +| FmpDevicePkg | :heavy_check_mark: |
> :heavy_check_mark: |
>
> +| IntelFsp2Pkg |
>
> +| IntelFsp2WrapperPkg |
>
> +| MdeModulePkg | :heavy_check_mark: |
> :heavy_check_mark: | DxeIpl dependency on ArmPkg,
> Depends on StandaloneMmPkg, Spell checking in audit
> mode
>
> +| MdePkg | :heavy_check_mark: |
> :heavy_check_mark: | Spell checking in audit mode
>
> +| NetworkPkg | :heavy_check_mark: |
> :heavy_check_mark: | Spell checking in audit mode
>
> +| OvmfPkg | SEE PACKAGE README | SEE
> PACKAGE README | Spell checking in audit mode
>
> +| PcAtChipsetPkg | :heavy_check_mark: |
> :heavy_check_mark: |
>
> +| SecurityPkg | :heavy_check_mark: |
> :heavy_check_mark: | Spell checking in audit mode
>
> +| ShellPkg | :heavy_check_mark: |
> :heavy_check_mark: | Spell checking in audit mode, 3
> modules are not being built by DSC
>
> +| SignedCapsulePkg |
>
> +| SourceLevelDebugPkg |
>
> +| StandaloneMmPkg |
>
> +| UefiCpuPkg | :heavy_check_mark: |
> :heavy_check_mark: | Spell checking in audit mode, 2
> binary modules not being built by DSC
>
> +| UefiPayloadPkg |
>
> +| UnitTestFrameworkPkg | :heavy_check_mark: |
> :heavy_check_mark: |
>
>
>
> For more detailed status look at the test results of
> the latest CI run on the
>
> repo readme.
>
> @@ -88,7 +89,7 @@ that a few steps should be followed.
> Details of EDKII Tools can be found in the
>
> * VS 2017 or VS 2019
>
> * Windows SDK (for rc)
>
> * Windows WDK (for capsules)
>
> - * Ubuntu 16.04
>
> + * Ubuntu 18.04 or Fedora
>
> * GCC5
>
> * Easy to add more but this is the current state
>
> 2. Python 3.7.x or newer on path
>
> @@ -137,11 +138,31 @@ location makes more sense for the
> community.
>
>
>
> ### Module Inclusion Test - DscCompleteCheck
>
>
>
> -This test scans all available modules (via INF files)
> and compares them to the
>
> -package-level DSC file for the package each module is
> contained within. The test
>
> -considers it an error if any module does not appear in
> the `Components` section
>
> -of at least one package-level DSC (indicating that it
> would not be built if the
>
> -package were built).
>
> +This scans all INF files from a package and confirms
> they are
>
> +listed in the package level DSC file. The test
> considers it an error if any INF
>
> +does not appear in the `Components` section of the
> package-level DSC (indicating
>
> +that it would not be built if the package were built).
> This is critical because
>
> +much of the CI infrastructure assumes that all modules
> will be listed in the DSC
>
> +and compiled.
>
> +
>
> +This test will ignore INFs in the following cases:
>
> +
>
> +1. When `MODULE_TYPE` = `HOST_APPLICATION`
>
> +2. When a Library instance **only** supports the
> `HOST_APPLICATION` environment
>
> +
>
> +### Host Module Inclusion Test -
> HostUnitTestDscCompleteCheck
>
> +
>
> +This test scans all INF files from a package for those
> related to host
>
> +based unit tests and confirms they are listed in the
> unit test DSC file for the package.
>
> +The test considers it an error if any INF meeting the
> requirements does not appear
>
> +in the `Components` section of the unit test DSC. This
> is critical because
>
> +much of the CI infrastructure assumes that modules
> will be listed in the DSC
>
> +and compiled.
>
> +
>
> +This test will only require INFs in the following
> cases:
>
> +
>
> +1. When `MODULE_TYPE` = `HOST_APPLICATION`
>
> +2. When a Library instance explicitly supports the
> `HOST_APPLICATION` environment
>
>
>
> ### Code Compilation Test - CompilerPlugin
>
>
>
> @@ -150,6 +171,46 @@ all package-level DSCs were built,
> the Code Compilation Test simply runs through
>
> and builds every package-level DSC on every toolchain
> and for every architecture
>
> that is supported. Any module that fails to build is
> considered an error.
>
>
>
> +### Host Unit Test Compilation and Run Test -
> HostUnitTestCompilerPlugin
>
> +
>
> +A test that compiles the dsc for host based unit test
> apps.
>
> +On Windows this will also enable a build plugin to
> execute that will run the unit tests and verify the
> results.
>
> +
>
> +These tools will be invoked on any CI
>
> +pass that includes the NOOPT target. In order for
> these tools to do their job,
>
> +the package and tests must be configured in a
> particular way...
>
> +
>
> +#### Including Host-Based Tests in the Package YAML
>
> +
>
> +For example, looking at the `MdeModulePkg.ci.yaml`
> config file, there are two
>
> +config options that control HostBased test behavior:
>
> +
>
> +```json
>
> + ## options defined
> .pytool/Plugin/HostUnitTestCompilerPlugin
>
> + "HostUnitTestCompilerPlugin": {
>
> + "DscPath": "Test/MdeModulePkgHostTest.dsc"
>
> + },
>
> +```
>
> +
>
> +This option tell the test builder to run. The test
> builder needs to know which
>
> +modules in this package are host-based tests, so that
> DSC path is provided.
>
> +
>
> +#### Configuring the HostBased DSC
>
> +
>
> +The HostBased DSC for `MdeModulePkg` is located at
>
> +`MdeModulePkg/Test/MdeModulePkgHostTest.dsc`.
>
> +
>
> +To add automated host-based unit test building to a
> new package, create a
>
> +similar DSC. The new DSC should make sure to have the
> `NOOPT` BUILD_TARGET
>
> +and should include the line:
>
> +
>
> +```
>
> +!include
> UnitTestFrameworkPkg/UnitTestFrameworkPkgHost.dsc.inc
>
> +```
>
> +
>
> +All of the modules that are included in the
> `Components` section of this
>
> +DSC should be of type HOST_APPLICATION.
>
> +
>
> ### GUID Uniqueness Test - GuidCheck
>
>
>
> This test works on the collection of all packages
> rather than an individual
>
> @@ -207,6 +268,8 @@ few standard scopes.
>
> | global-nix | edk2_invocable++
> | Running on Linux based OS |
>
> | edk2-build |
> | This indicates that an invocable is building EDK2
> based UEFI code |
>
> | cibuild | set in .pytool/CISettings.py
> | Suggested target for edk2 continuous integration
> builds. Tools used for CiBuilds can use this scope.
> Example: asl compiler |
>
> +| host-based-test | set in .pytool/CISettings.py
> | Turns on the host based tests and plugin |
>
> +| host-test-win | set in .pytool/CISettings.py
> | Enables the host based test runner for Windows |
>
>
>
> ## Future investments
>
>
>
> diff --git a/UnitTestFrameworkPkg/ReadMe.md
> b/UnitTestFrameworkPkg/ReadMe.md
>
> index 7296f0a45c5f..64386941cb4b 100644
>
> --- a/UnitTestFrameworkPkg/ReadMe.md
>
> +++ b/UnitTestFrameworkPkg/ReadMe.md
>
> @@ -58,7 +58,7 @@ header for the `UnitTestLib` is
> located in `MdePkg`, so you shouldn't need to de
>
> packages. As long as your DSC file knows where to find
> the lib implementation that you want to use,
>
> you should be good to go.
>
>
>
> -See this example in 'SampleUnitTestApp.inf'...
>
> +See this example in 'SampleUnitTestUefiShell.inf'...
>
>
>
> ```
>
> [Packages]
>
> @@ -72,6 +72,14 @@ See this example in
> 'SampleUnitTestApp.inf'...
>
> PrintLib
>
> ```
>
>
>
> +Also, if you want you test to automatically be picked
> up by the Test Runner plugin, you will need
>
> +to make sure that the module `BASE_NAME` contains the
> word `Test`...
>
> +
>
> +```
>
> +[Defines]
>
> + BASE_NAME = SampleUnitTestUefiShell
>
> +```
>
> +
>
> ### Requirements - Code
>
>
>
> Not to state the obvious, but let's make sure we have
> the following include before getting too far along...
>
> @@ -221,9 +229,11 @@ https://api.cmocka.org/
>
>
>
> ## Development
>
>
>
> -When using the EDK2 Pytools for CI testing, the host-
> based unit tests will be built and run on any build
> that includes the `NOOPT` build target.
>
> +When using the EDK2 Pytools for CI testing, the host-
> based unit tests will be built and run on any build
> that includes
>
> +the `NOOPT` build target.
>
>
>
> -If you are trying to iterate on a single test, a
> convenient pattern is to build only that test module.
> For example, the following command will build only the
> SafeIntLib host-based test from the MdePkg...
>
> +If you are trying to iterate on a single test, a
> convenient pattern is to build only that test module.
> For example,
>
> +the following command will build only the SafeIntLib
> host-based test from the MdePkg...
>
>
>
> ```bash
>
> stuart_ci_build -c .pytool/CISettings.py
> TOOL_CHAIN_TAG=VS2017 -p MdePkg -t NOOPT
> BUILDMODULE=MdePkg/Test/UnitTest/Library/BaseSafeIntLib
> /TestBaseSafeIntLib.inf
>
> @@ -250,8 +260,72 @@ reporting lib. This isn't
> currently possible with host-based. Only the assertion
>
>
>
> We will continue trying to make these as similar as
> possible.
>
>
>
> +## Unit Test Location/Layout Rules
>
> +
>
> +Code/Test | Location
>
> +--------- | --------
>
> +Host-Based Unit Tests for a Library/Protocol/PPI/GUID
> Interface | If what's being tested is an interface
> (e.g. a library with a public header file, like
> DebugLib), the test should be scoped to the parent
> package.<br/>Example:
> `MdePkg/Test/UnitTest/[Library/Protocol/Ppi/Guid]/`<br/
> ><br/>A real-world example of this is the
> BaseSafeIntLib test in
> MdePkg.<br/>`MdePkg/Test/UnitTest/Library/BaseSafeIntLi
> b/TestBaseSafeIntLibHost.inf`
>
> +Host-Based Unit Tests for a Library/Driver
> (PEI/DXE/SMM) implementation | If what's being tested
> is a specific implementation (e.g.
> BaseDebugLibSerialPort for DebugLib), the test should
> be scoped to the implementation directory itself, in a
> UnitTest subdirectory.<br/><br/>Module Example:
> `MdeModulePkg/Universal/EsrtFmpDxe/UnitTest/`<br/>Libra
> ry Example: `MdePkg/Library/BaseMemoryLib/UnitTest/`
>
> +Host-Based Tests for a Functionality or Feature | If
> you're writing a functional test that operates at the
> module level (i.e. if it's more than a single file or
> library), the test should be located in the package-
> level Tests directory under the HostFuncTest
> subdirectory.<br/>For example, if you were writing a
> test for the entire FMP Device Framework, you might put
> your test
> in:<br/>`FmpDevicePkg/Test/HostFuncTest/FmpDeviceFramew
> ork`<br/><br/>If the feature spans multiple packages,
> it's location should be determined by the package
> owners related to the feature.
>
> +Non-Host-Based (PEI/DXE/SMM/Shell) Tests for a
> Functionality or Feature | Similar to Host-Based, if
> the feature is in one package, should be located in the
> `*Pkg/Test/[Shell/Dxe/Smm/Pei]Test`
> directory.<br/><br/>If the feature spans multiple
> packages, it's location should be determined by the
> package owners related to the feature.<br/><br/>USAGE
> EXAMPLES<br/>PEI Example: MP_SERVICE_PPI. Or check MTRR
> configuration in a notification function.<br/> SMM
> Example: a test in a protocol callback function. (It is
> different with the solution that
> SmmAgent+ShellApp)<br/>DXE Example: a test in a UEFI
> event call back to check SPI/SMRAM status. <br/> Shell
> Example: the SMM handler audit test has a shell-based
> app that interacts with an SMM handler to get
> information. The SMM paging audit test gathers
> information about both DXE and SMM. And the SMM paging
> functional test actually forces errors into SMM via a
> DXE driver.
>
> +
>
> +### Example Directory Tree
>
> +
>
> +```text
>
> +<PackageName>Pkg/
>
> + ComponentY/
>
> + ComponentY.inf
>
> + ComponentY.c
>
> + UnitTest/
>
> + ComponentYHostUnitTest.inf # Host-Based
> Test for Driver Module
>
> + ComponentYUnitTest.c
>
> +
>
> + Library/
>
> + GeneralPurposeLibBase/
>
> + ...
>
> +
>
> + GeneralPurposeLibSerial/
>
> + ...
>
> +
>
> + SpecificLibDxe/
>
> + SpecificLibDxe.c
>
> + SpecificLibDxe.inf
>
> + UnitTest/ # Host-Based Test
> for Specific Library Implementation
>
> + SpecificLibDxeHostUnitTest.c
>
> + SpecificLibDxeHostUnitTest.inf
>
> + Test/
>
> + <Package>HostTest.dsc # Host-Based
> Test Apps
>
> + UnitTest/
>
> + InterfaceX
>
> + InterfaceXHostUnitTest.inf # Host-Based App
> (should be in Test/<Package>HostTest.dsc)
>
> + InterfaceXPeiUnitTest.inf # PEIM Target-
> Based Test (if applicable)
>
> + InterfaceXDxeUnitTest.inf # DXE Target-
> Based Test (if applicable)
>
> + InterfaceXSmmUnitTest.inf # SMM Target-
> Based Test (if applicable)
>
> + InterfaceXShellUnitTest.inf # Shell App
> Target-Based Test (if applicable)
>
> + InterfaceXUnitTest.c # Test Logic
>
> +
>
> + GeneralPurposeLib/ # Host-Based
> Test for any implementation of GeneralPurposeLib
>
> + GeneralPurposeLibTest.c
>
> + GeneralPurposeLibHostUnitTest.inf
>
> +
>
> + <Package>Pkg.dsc # Standard Modules and any
> Target-Based Test Apps (including in Test/)
>
> +
>
> +```
>
> +
>
> +### Future Locations in Consideration
>
> +
>
> +We don't know if these types will exist or be
> applicable yet, but if you write a support library or
> module that matches the following, please make sure
> they live in the correct place.
>
> +
>
> +Code/Test | Location
>
> +--------- | --------
>
> +Host-Based Library Implementations |
> Host-Based Implementations of common libraries (eg.
> MemoryAllocationLibHost) should live in the same
> package that declares the library interface in its .DEC
> file in the `*Pkg/HostLibrary` directory. Should have
> 'Host' in the name.
>
> +Host-Based Mocks and Stubs | Mock and Stub libraries
> should live in the `UefiTestFrameworkPkg/StubLibrary`
> with either 'Mock' or 'Stub' in the library name.
>
> +
>
> +### If still in doubt...
>
> +
>
> +Hop on GitHub and ask @corthon, @mdkinney, or
> @spbrogan. ;)
>
> +
>
> ## Copyright
>
>
>
> Copyright (c) Microsoft Corporation.
>
> SPDX-License-Identifier: BSD-2-Clause-Patent
>
> -
>
> diff --git
> a/UnitTestFrameworkPkg/UnitTestFrameworkPkg.ci.yaml
> b/UnitTestFrameworkPkg/UnitTestFrameworkPkg.ci.yaml
>
> index 01648595055e..51e172537f8a 100644
>
> --- a/UnitTestFrameworkPkg/UnitTestFrameworkPkg.ci.yaml
>
> +++ b/UnitTestFrameworkPkg/UnitTestFrameworkPkg.ci.yaml
>
> @@ -59,7 +59,7 @@
>
> "SpellCheck": {
>
> "AuditOnly": False, # Fails test but
> run in AuditOnly mode to collect log
>
> "IgnoreFiles": [ # use gitignore
> syntax to ignore errors in matching files
>
> - "/Library/CmockaLib/cmocka/**/*.*" # not
> going to spell check a submodule
>
> + "Library/CmockaLib/cmocka/**/*.*" # not
> going to spell check a submodule
>
> ],
>
> "ExtendWords": [ # words to extend
> to the dictionary for this package
>
> "cmocka",
>
> @@ -68,7 +68,10 @@
>
> "pytool",
>
> "pytools",
>
> "NOFAILURE",
>
> - "DHAVE" # build flag for cmocka in the INF
>
> + "DHAVE", # build flag for cmocka in the
> INF
>
> + "corthon", # Contact GitHub account
> in Readme
>
> + "mdkinney", # Contact GitHub account
> in Readme
>
> + "spbrogan" # Contact GitHub account
> in Readme
>
> ],
>
> "IgnoreStandardPaths": [], # Standard Plugin
> defined paths that should be ignore
>
> "AdditionalIncludePaths": [] # Additional
> paths to spell check (wildcards supported)
>
> --
>
> 2.26.2.windows.1.8.g01c50adf56.20200515075929
>
>
-=-=-=-=-=-=-=-=-=-=-=-
Groups.io Links: You receive all messages sent to this group.
View/Reply Online (#60466): https://edk2.groups.io/g/devel/message/60466
Mute This Topic: https://groups.io/mt/74549174/1813853
Group Owner: devel+owner at edk2.groups.io
Unsubscribe: https://edk2.groups.io/g/devel/unsub [edk2-devel-archive at redhat.com]
-=-=-=-=-=-=-=-=-=-=-=-
More information about the edk2-devel-archive
mailing list