[edk2-devel] [PATCH edk2-platforms v1 3/6] .pytool/Plugin: Add CI plugins
PierreGondois
pierre.gondois at arm.com
Wed Jun 23 08:02:06 UTC 2021
From: Pierre Gondois <Pierre.Gondois at arm.com>
To enable CI support of the tianocore/edk2-platforms repository,
add a .pytool directory containing the following files:
- .pytool/CISettings.py
- .pytool/Readme.md
These files are largely inspired from the same files available in
the edk2 repository. The .pytool/Plugin/* files containing the
CI tests to run are not copied. edk2-platforms will rely on the
edk2basetools python package and on the edk2 python files, as
edk2 is imported as a submodule of edk2-platforms.
Cc: Sean Brogan <sean.brogan at microsoft.com>
Cc: Bret Barkelew <Bret.Barkelew at microsoft.com>
Cc: Michael D Kinney <michael.d.kinney at intel.com>
Cc: Liming Gao <gaoliming at byosoft.com.cn>
Cc: Sami Mujawar <sami.mujawar at arm.com>
Signed-off-by: Sami Mujawar <sami.mujawar at arm.com>
Signed-off-by: Pierre Gondois <Pierre.Gondois at arm.com>
---
.pytool/CISettings.py | 185 ++++++++++++++++++++++++++++
.pytool/Readme.md | 271 ++++++++++++++++++++++++++++++++++++++++++
2 files changed, 456 insertions(+)
create mode 100644 .pytool/CISettings.py
create mode 100644 .pytool/Readme.md
diff --git a/.pytool/CISettings.py b/.pytool/CISettings.py
new file mode 100644
index 000000000000..18604646030f
--- /dev/null
+++ b/.pytool/CISettings.py
@@ -0,0 +1,185 @@
+# @file
+#
+# Copyright (c) Microsoft Corporation.
+# Copyright (c) 2020, Hewlett Packard Enterprise Development LP. All rights reserved.<BR>
+# Copyright (c) 2020 - 2021, ARM Limited. All rights reserved.<BR>
+# SPDX-License-Identifier: BSD-2-Clause-Patent
+##
+import os
+import logging
+import edk2basetools
+
+from edk2toolext.environment import shell_environment
+from edk2toolext.invocables.edk2_ci_build import CiBuildSettingsManager
+from edk2toolext.invocables.edk2_setup import SetupSettingsManager, RequiredSubmodule
+from edk2toolext.invocables.edk2_update import UpdateSettingsManager
+from edk2toolext.invocables.edk2_pr_eval import PrEvalSettingsManager
+from edk2toollib.utility_functions import GetHostInfo
+
+
+class Settings(CiBuildSettingsManager, UpdateSettingsManager, SetupSettingsManager, PrEvalSettingsManager):
+
+ def __init__(self):
+ self.ActualPackages = []
+ self.ActualTargets = []
+ self.ActualArchitectures = []
+ self.ActualToolChainTag = ""
+ self.ActualScopes = None
+
+ # ####################################################################################### #
+ # Extra CmdLine configuration #
+ # ####################################################################################### #
+
+ def AddCommandLineOptions(self, parserObj):
+ pass
+ def RetrieveCommandLineOptions(self, args):
+ pass
+
+ # ####################################################################################### #
+ # Default Support for this Ci Build #
+ # ####################################################################################### #
+
+ def GetPackagesSupported(self):
+ ''' return iterable of edk2 packages supported by this build.
+ These should be edk2 workspace relative paths '''
+ return (
+ "JunoPkg",
+ "VExpressPkg"
+ )
+
+ def GetArchitecturesSupported(self):
+ ''' return iterable of edk2 architectures supported by this build '''
+ return (
+ "IA32",
+ "X64",
+ "ARM",
+ "AARCH64",
+ "RISCV64")
+
+ def GetTargetsSupported(self):
+ ''' return iterable of edk2 target tags supported by this build '''
+ return ("DEBUG", "RELEASE", "NO-TARGET", "NOOPT")
+
+ # ####################################################################################### #
+ # Verify and Save requested Ci Build Config #
+ # ####################################################################################### #
+
+ def SetPackages(self, list_of_requested_packages):
+ ''' Confirm the requested package list is valid and configure SettingsManager
+ to build the requested packages.
+
+ Raise UnsupportedException if a requested_package is not supported
+ '''
+ unsupported = set(list_of_requested_packages) - \
+ set(self.GetPackagesSupported())
+ if(len(unsupported) > 0):
+ logging.critical(
+ "Unsupported Package Requested: " + " ".join(unsupported))
+ raise Exception("Unsupported Package Requested: " +
+ " ".join(unsupported))
+ self.ActualPackages = list_of_requested_packages
+
+ def SetArchitectures(self, list_of_requested_architectures):
+ ''' Confirm the requests architecture list is valid and configure SettingsManager
+ to run only the requested architectures.
+
+ Raise Exception if a list_of_requested_architectures is not supported
+ '''
+ unsupported = set(list_of_requested_architectures) - \
+ set(self.GetArchitecturesSupported())
+ if(len(unsupported) > 0):
+ logging.critical(
+ "Unsupported Architecture Requested: " + " ".join(unsupported))
+ raise Exception(
+ "Unsupported Architecture Requested: " + " ".join(unsupported))
+ self.ActualArchitectures = list_of_requested_architectures
+
+ def SetTargets(self, list_of_requested_target):
+ ''' Confirm the request target list is valid and configure SettingsManager
+ to run only the requested targets.
+
+ Raise UnsupportedException if a requested_target is not supported
+ '''
+ unsupported = set(list_of_requested_target) - \
+ set(self.GetTargetsSupported())
+ if(len(unsupported) > 0):
+ logging.critical(
+ "Unsupported Targets Requested: " + " ".join(unsupported))
+ raise Exception("Unsupported Targets Requested: " +
+ " ".join(unsupported))
+ self.ActualTargets = list_of_requested_target
+
+ # ####################################################################################### #
+ # Actual Configuration for Ci Build #
+ # ####################################################################################### #
+
+ def GetActiveScopes(self):
+ ''' return tuple containing scopes that should be active for this process '''
+ if self.ActualScopes is None:
+ scopes = ("cibuild", "edk2-build", "host-based-test")
+
+ self.ActualToolChainTag = shell_environment.GetBuildVars().GetValue("TOOL_CHAIN_TAG", "")
+
+ is_linux = GetHostInfo().os.upper() == "LINUX"
+ scopes += ('pipbuild-unix',) if is_linux else ('pipbuild-win',)
+
+ if is_linux and self.ActualToolChainTag.upper().startswith("GCC"):
+ if "AARCH64" in self.ActualArchitectures:
+ scopes += ("gcc_aarch64_linux",)
+ if "ARM" in self.ActualArchitectures:
+ scopes += ("gcc_arm_linux",)
+ if "RISCV64" in self.ActualArchitectures:
+ scopes += ("gcc_riscv64_unknown",)
+ self.ActualScopes = scopes
+ return self.ActualScopes
+
+ def GetRequiredSubmodules(self):
+ ''' return iterable containing RequiredSubmodule objects.
+ If no RequiredSubmodules return an empty iterable
+ '''
+ rs = []
+ rs.append(RequiredSubmodule(
+ "edk2", True))
+ rs.append(RequiredSubmodule(
+ "Silicon/RISC-V/ProcessorPkg/Library/RiscVOpensbiLib/opensbi", False))
+ return rs
+
+ def GetName(self):
+ return "Edk2-platforms"
+
+ def GetDependencies(self):
+ return [
+ ]
+
+ def GetPackagesPath(self):
+ ''' Return a list of workspace relative paths that should be mapped as edk2 PackagesPath '''
+ edk2_platforms_path = os.path.dirname(os.path.dirname(os.path.abspath(__file__)))
+ return [
+ edk2_platforms_path,
+ os.path.join(edk2_platforms_path, "Platform", "ARM"),
+ os.path.join(edk2_platforms_path, "edk2")
+ ]
+
+ def GetWorkspaceRoot(self):
+ ''' get WorkspacePath '''
+ return os.path.dirname(os.path.dirname(os.path.abspath(__file__)))
+
+ def FilterPackagesToTest(self, changedFilesList: list, potentialPackagesList: list) -> list:
+ ''' Filter potential packages to test based on changed files. '''
+ build_these_packages = []
+ possible_packages = potentialPackagesList.copy()
+ for f in changedFilesList:
+ # split each part of path for comparison later
+ nodes = f.split("/")
+
+ # python file change in .pytool folder causes building all
+ if f.endswith(".py") and ".pytool" in nodes:
+ build_these_packages = possible_packages
+ break
+
+ # BaseTools files that might change the build
+ if "BaseTools" in nodes:
+ if os.path.splitext(f) not in [".txt", ".md"]:
+ build_these_packages = possible_packages
+ break
+ return build_these_packages
diff --git a/.pytool/Readme.md b/.pytool/Readme.md
new file mode 100644
index 000000000000..5b3d9679d7f2
--- /dev/null
+++ b/.pytool/Readme.md
@@ -0,0 +1,271 @@
+# Edk2-platforms Continuous Integration
+
+## Basic Status
+
+| Package | Windows VS2019 (IA32/X64)| Ubuntu GCC (IA32/X64/ARM/AARCH64) | Known Issues |
+| :---- | :----- | :---- | :--- |
+| Platfrom/ARM/JunoPkg | | :heavy_check_mark: | Spell checking in audit mode. CompilerCheck disabled (need a PlatformCI).
+
+For more detailed status look at the test results of the latest CI run on the
+repo readme.
+
+## Background
+
+This Continuous integration and testing infrastructure leverages the TianoCore EDKII Tools PIP modules:
+[library](https://pypi.org/project/edk2-pytool-library/) and
+[extensions](https://pypi.org/project/edk2-pytool-extensions/) (with repos
+located [here](https://github.com/tianocore/edk2-pytool-library) and
+[here](https://github.com/tianocore/edk2-pytool-extensions)).
+
+The primary execution flows can be found in the
+`.azurepipelines/Windows-VS2019.yml` and `.azurepipelines/Ubuntu-GCC5.yml`
+files. These YAML files are consumed by the Azure Dev Ops Build Pipeline and
+dictate what server resources should be used, how they should be configured, and
+what processes should be run on them. An overview of this schema can be found
+[here](https://docs.microsoft.com/en-us/azure/devops/pipelines/yaml-schema?view=azure-devops&tabs=schema).
+
+Inspection of these files reveals the EDKII Tools commands that make up the
+primary processes for the CI build: 'stuart_setup', 'stuart_update', and
+'stuart_ci_build'. These commands come from the EDKII Tools PIP modules and are
+configured as described below. More documentation on the tools can be
+found [here](https://github.com/tianocore/edk2-pytool-extensions/blob/master/docs/using.md)
+and [here](https://github.com/tianocore/edk2-pytool-extensions/blob/master/docs/features/feature_invocables.md).
+
+## Configuration
+
+Configuration of the CI process consists of (in order of precedence):
+
+* command-line arguments passed in via the Pipeline YAML
+* a per-package configuration file (e.g. `<package-name>.ci.yaml`) that is
+ detected by the CI system in EDKII Tools.
+* a global configuration Python module (e.g. `CISetting.py`) passed in via the
+ command-line
+
+The global configuration file is described in
+[this readme](https://github.com/tianocore/edk2-pytool-extensions/blob/master/docs/usability/using_settings_manager.md)
+from the EDKII Tools documentation. This configuration is written as a Python
+module so that decisions can be made dynamically based on command line
+parameters and codebase state.
+
+The per-package configuration file can override most settings in the global
+configuration file, but is not dynamic. This file can be used to skip or
+customize tests that may be incompatible with a specific package. Each test generally requires
+per package configuration which comes from this file.
+
+## Running CI locally
+
+The EDKII Tools environment (and by extension the ci) is designed to support
+easily and consistently running locally and in a cloud ci environment. To do
+that a few steps should be followed. Details of EDKII Tools can be found in the
+[docs folder here](https://github.com/tianocore/edk2-pytool-extensions/tree/master/docs)
+
+### Prerequisets
+
+1. A supported toolchain (others might work but this is what is tested and validated)
+ * Windows 10:
+ * VS 2017 or VS 2019
+ * Windows SDK (for rc)
+ * Windows WDK (for capsules)
+ * Ubuntu 18.04 or Fedora
+ * GCC5
+ * Easy to add more but this is the current state
+2. Python 3.7.x or newer on path
+3. git on path
+4. Recommended to setup and activate a python virtual environment
+5. Install the requirements `pip install --upgrade pip-requirements.txt`
+
+### Running CI
+
+1. clone your edk2-platforms repo
+2. Activate your python virtual environment in cmd window
+3. Get code dependencies (done only when submodules change)
+ * `stuart_setup -c .pytool/CISettings.py TOOL_CHAIN_TAG=<your tag here>`
+4. Update other dependencies (done more often)
+ * `stuart_update -c .pytool/CISettings.py TOOL_CHAIN_TAG=<your tag here>`
+5. Run CI build (--help will give you options)
+ * `stuart_ci_build -c .pytool/CISettings.py TOOL_CHAIN_TAG=<your tag here>`
+ * -p <pkg1,pkg2,pkg3> : To build only certain packages use a CSV list
+ * -a <arch1,arch2,arch3>: To run only certain architectures use a CSV list
+ * -t <target1,target2>: To run only tests related to certain targets use a
+ CSV list
+ * By default all tests are opted in. Then given a package.ci.yaml file those
+ tests can be configured for a package. Finally setting the check to the
+ value `skip` will skip that plugin. Examples:
+ * `CompilerPlugin=skip` skip the build test
+ * `GuidCheck=skip` skip the Guid check
+ * `SpellCheck=skip` skip the spell checker
+ * etc
+6. Detailed reports and logs per package are captured in the `Build` directory
+
+## Current PyTool Test Capabilities
+
+All CI tests are instances of EDKII Tools plugins. Documentation on the plugin
+system can be found [here](https://github.com/tianocore/edk2-pytool-extensions/blob/master/docs/usability/using_plugin_manager.md)
+and [here](https://github.com/tianocore/edk2-pytool-extensions/blob/master/docs/features/feature_plugin_manager.md).
+Upon invocation, each plugin will be passed the path to the current package
+under test and a dictionary containing its targeted configuration, as assembled
+from the command line, per-package configuration, and global configuration.
+
+Note: CI plugins are considered unique from build plugins and helper plugins,
+even though some CI plugins may execute steps of a build.
+
+In the example, these plugins live alongside the code under test (in the
+`.pytool/Plugin` directory), but may be moved to the 'edk2-test' repo if that
+location makes more sense for the community.
+
+### Module Inclusion Test - DscCompleteCheck
+
+This scans all INF files from a package and confirms they are
+listed in the package level DSC file. The test considers it an error if any INF
+does not appear in the `Components` section of the package-level DSC (indicating
+that it would not be built if the package were built). This is critical because
+much of the CI infrastructure assumes that all modules will be listed in the DSC
+and compiled.
+
+This test will ignore INFs in the following cases:
+
+1. When `MODULE_TYPE` = `HOST_APPLICATION`
+2. When a Library instance **only** supports the `HOST_APPLICATION` environment
+
+### Host Module Inclusion Test - HostUnitTestDscCompleteCheck
+
+This test scans all INF files from a package for those related to host
+based unit tests and confirms they are listed in the unit test DSC file for the package.
+The test considers it an error if any INF meeting the requirements does not appear
+in the `Components` section of the unit test DSC. This is critical because
+much of the CI infrastructure assumes that modules will be listed in the DSC
+and compiled.
+
+This test will only require INFs in the following cases:
+
+1. When `MODULE_TYPE` = `HOST_APPLICATION`
+2. When a Library instance explicitly supports the `HOST_APPLICATION` environment
+
+### Code Compilation Test - CompilerPlugin
+
+Once the Module Inclusion Test has verified that all modules would be built if
+all package-level DSCs were built, the Code Compilation Test simply runs through
+and builds every package-level DSC on every toolchain and for every architecture
+that is supported. Any module that fails to build is considered an error.
+
+### Host Unit Test Compilation and Run Test - HostUnitTestCompilerPlugin
+
+A test that compiles the dsc for host based unit test apps.
+On Windows this will also enable a build plugin to execute that will run the unit tests and verify the results.
+
+These tools will be invoked on any CI
+pass that includes the NOOPT target. In order for these tools to do their job,
+the package and tests must be configured in a particular way...
+
+#### Including Host-Based Tests in the Package YAML
+
+For example, looking at the `MdeModulePkg.ci.yaml` config file, there are two
+config options that control HostBased test behavior:
+
+```json
+ ## options defined .pytool/Plugin/HostUnitTestCompilerPlugin
+ "HostUnitTestCompilerPlugin": {
+ "DscPath": "Test/MdeModulePkgHostTest.dsc"
+ },
+```
+
+This option tell the test builder to run. The test builder needs to know which
+modules in this package are host-based tests, so that DSC path is provided.
+
+#### Configuring the HostBased DSC
+
+The HostBased DSC for `MdeModulePkg` is located at
+`MdeModulePkg/Test/MdeModulePkgHostTest.dsc`.
+
+To add automated host-based unit test building to a new package, create a
+similar DSC. The new DSC should make sure to have the `NOOPT` BUILD_TARGET
+and should include the line:
+
+```
+!include UnitTestFrameworkPkg/UnitTestFrameworkPkgHost.dsc.inc
+```
+
+All of the modules that are included in the `Components` section of this
+DSC should be of type HOST_APPLICATION.
+
+### GUID Uniqueness Test - GuidCheck
+
+This test works on the collection of all packages rather than an individual
+package. It looks at all FILE_GUIDs and GUIDs declared in DEC files and ensures
+that they are unique for the codebase. This prevents, for example, accidental
+duplication of GUIDs when using an existing INF as a template for a new module.
+
+### Cross-Package Dependency Test - DependencyCheck
+
+This test compares the list of all packages used in INFs files for a given
+package against a list of "allowed dependencies" in plugin configuration for
+that package. Any module that depends on a disallowed package will cause a test
+failure.
+
+### Library Declaration Test - LibraryClassCheck
+
+This test scans at all library header files found in the `Library` folders in
+all of the package's declared include directories and ensures that all files
+have a matching LibraryClass declaration in the DEC file for the package. Any
+missing declarations will cause a failure.
+
+### Invalid Character Test - CharEncodingCheck
+
+This test scans all files in a package to make sure that there are no invalid
+Unicode characters that may cause build errors in some character
+sets/localizations.
+
+### Spell Checking - cspell
+
+This test runs a spell checker on all files within the package. This is done
+using the NodeJs cspell tool. For details check `.pytool/Plugin/SpellCheck`.
+For this plugin to run during ci you must install nodejs and cspell and have
+both available to the command line when running your CI.
+
+Install
+
+* Install nodejs from https://nodejs.org/en/
+* Install cspell
+ 1. Open cmd prompt with access to node and npm
+ 2. Run `npm install -g cspell`
+
+ More cspell info: https://github.com/streetsidesoftware/cspell
+
+### License Checking - LicenseCheck
+
+Scans all new added files in a package to make sure code is contributed under
+BSD-2-Clause-Patent.
+
+### Ecc tool - EccCheck
+
+Run the Ecc tool on the package. The Ecc tool is available in the BaseTools
+package. It checks that the code complies to the EDKII coding standard.
+
+## PyTool Scopes
+
+Scopes are how the PyTool ext_dep, path_env, and plugins are activated. Meaning
+that if an invocable process has a scope active then those ext_dep and path_env
+will be active. To allow easy integration of PyTools capabilities there are a
+few standard scopes.
+
+| Scope | Invocable | Description |
+| :---- | :----- | :---- |
+| global | edk2_invocable++ - should be base_abstract_invocable | Running an invocables |
+| global-win | edk2_invocable++ | Running on Microsoft Windows |
+| global-nix | edk2_invocable++ | Running on Linux based OS |
+| edk2-build | | This indicates that an invocable is building EDK2 based UEFI code |
+| cibuild | set in .pytool/CISettings.py | Suggested target for edk2 continuous integration builds. Tools used for CiBuilds can use this scope. Example: asl compiler |
+| host-based-test | set in .pytool/CISettings.py | Turns on the host based tests and plugin |
+| host-test-win | set in .pytool/CISettings.py | Enables the host based test runner for Windows |
+
+## Future investments
+
+* PatchCheck tests as plugins
+* MacOS/xcode support
+* Clang/LLVM support
+* Visual Studio AARCH64 and ARM support
+* BaseTools C tools CI/PR and binary release process
+* BaseTools Python tools CI/PR process
+* Extensible private/closed source platform reporting
+* UEFI SCTs
+* Other automation
--
2.17.1
-=-=-=-=-=-=-=-=-=-=-=-
Groups.io Links: You receive all messages sent to this group.
View/Reply Online (#76917): https://edk2.groups.io/g/devel/message/76917
Mute This Topic: https://groups.io/mt/83733298/1813853
Group Owner: devel+owner at edk2.groups.io
Unsubscribe: https://edk2.groups.io/g/devel/unsub [edk2-devel-archive at redhat.com]
-=-=-=-=-=-=-=-=-=-=-=-
More information about the edk2-devel-archive
mailing list