<a name="_MailOriginal"></a>RFC Edk2-ToolEnv creation <span style="mso-ascii-font-family: Calibri; mso-ascii-theme-font: minor-latin; mso-hansi-font-family: Calibri; mso-hansi-theme-font: minor-latin; mso-bidi-font-family: 'Times New Roman'; mso-bidi-theme-font: minor-bidi;"> Create a new tianocore owned repository to host python code to support an extensible, pluggable, rich environment. This environment has command line interfaces to support building a product, building CI, running tests, and downloading dependencies. This environment also provides the building blocks for developers to write their own tools to launch in the environment and leverage the capabilities provided by the environment. The unique capabilities provided help support building products with multiple repositories and having each repository contribute/plugin to the build process in a scalable way. The environment will scan the files in the code tree (multiple repos) and discover plugins, dependencies, path adjustments, environment variable settings, etc. This provides easy methods for common repos to share build tools/steps. Inclusion of this package and dependency management should be managed using Pip/Pypi. To start this is a supplemental package and is not required to be used for edk2 builds. <span style="mso-ascii-font-family: Calibri; mso-ascii-theme-font: minor-latin; mso-hansi-font-family: Calibri; mso-hansi-theme-font: minor-latin; mso-bidi-font-family: 'Times New Roman'; mso-bidi-theme-font: minor-bidi;"> <span style="mso-ascii-font-family: Calibri; mso-ascii-theme-font: minor-latin; mso-hansi-font-family: Calibri; mso-hansi-theme-font: minor-latin; mso-bidi-font-family: 'Times New Roman'; mso-bidi-theme-font: minor-bidi;">More details below but much of this content is coming from combining two existing repos <ol style="margin-top: 0in;" start="1" type="1"> <li class="MsoListParagraphCxSpFirst" style="margin-bottom: 8.0pt; margin-left: 0in; mso-add-space: auto; line-height: 105%; mso-list: l0 level1 lfo10;"><a href="https://github.com/microsoft/mu_pip_environment">https://github.com/microsoft/mu_pip_environment</a></li> <li class="MsoListParagraphCxSpLast" style="margin-bottom: 8.0pt; margin-left: 0in; mso-add-space: auto; line-height: 105%; mso-list: l0 level1 lfo10;"><a href="https://github.com/microsoft/mu_pip_build">https://github.com/microsoft/mu_pip_build</a></li> </ol> This RFC is part 2 of content from design review: <a href="https://edk2.groups.io/g/devel/files/Designs/2019/0418/2019-04-18%20Microsoft%20-%20Build%20Tools%20-%20Design%20Review%20.pdf">https://edk2.groups.io/g/devel/files/Designs/2019/0418/2019-04-18%20Microsoft%20-%20Build%20Tools%20-%20Design%20Review%20.pdf</a> <span style="mso-ascii-font-family: Calibri; mso-ascii-theme-font: minor-latin; mso-hansi-font-family: Calibri; mso-hansi-theme-font: minor-latin; mso-bidi-font-family: 'Times New Roman'; mso-bidi-theme-font: minor-bidi;"> Examples of content here <ol style="margin-top: 0in;" start="1" type="1"> <li class="MsoListParagraphCxSpFirst" style="margin-bottom: 8.0pt; margin-left: 0in; mso-add-space: auto; line-height: 105%; mso-list: l1 level1 lfo1;">CI build support - <a href="https://github.com/microsoft/mu_pip_build/blob/master/MuBuild/MuBuild.py">https://github.com/microsoft/mu_pip_build/blob/master/MuBuild/MuBuild.py</a></li> <li class="MsoListParagraphCxSpMiddle" style="margin-bottom: 8.0pt; margin-left: 0in; mso-add-space: auto; line-height: 105%; mso-list: l1 level1 lfo1;">Binary Dependency resolution</li> <ol style="margin-top: 0in;" start="1" type="a"> <li class="MsoListParagraphCxSpMiddle" style="margin-bottom: 8.0pt; margin-left: 0in; mso-add-space: auto; line-height: 105%; mso-list: l1 level2 lfo1;">Nuget - <a href="https://github.com/microsoft/mu_pip_environment/blob/master/MuEnvironment/ExternalDependencies.py">https://github.com/microsoft/mu_pip_environment/blob/master/MuEnvironment/ExternalDependencies.py</a></li> <li class="MsoListParagraphCxSpMiddle" style="margin-bottom: 8.0pt; margin-left: 0in; mso-add-space: auto; line-height: 105%; mso-list: l1 level2 lfo1;">GitHub Releases - WIP</li> <li class="MsoListParagraphCxSpMiddle" style="margin-bottom: 8.0pt; margin-left: 0in; mso-add-space: auto; line-height: 105%; mso-list: l1 level2 lfo1;">URL dependency - WIP</li> </ol> <li class="MsoListParagraphCxSpMiddle" style="margin-bottom: 8.0pt; margin-left: 0in; mso-add-space: auto; line-height: 105%; mso-list: l1 level1 lfo1;">Logging </li> <ol style="margin-top: 0in;" start="1" type="a"> <li class="MsoListParagraphCxSpMiddle" style="margin-bottom: 8.0pt; margin-left: 0in; mso-add-space: auto; line-height: 105%; mso-list: l1 level2 lfo1;">File logger</li> <li class="MsoListParagraphCxSpMiddle" style="margin-bottom: 8.0pt; margin-left: 0in; mso-add-space: auto; line-height: 105%; mso-list: l1 level2 lfo1;">Markdown logger</li> <li class="MsoListParagraphCxSpMiddle" style="margin-bottom: 8.0pt; margin-left: 0in; mso-add-space: auto; line-height: 105%; mso-list: l1 level2 lfo1;">In Memory loggers - For tool parsing</li> <li class="MsoListParagraphCxSpMiddle" style="margin-bottom: 8.0pt; margin-left: 0in; mso-add-space: auto; line-height: 105%; mso-list: l1 level2 lfo1;">Console loggers with colors</li> </ol> <li class="MsoListParagraphCxSpMiddle" style="margin-bottom: 8.0pt; margin-left: 0in; mso-add-space: auto; line-height: 105%; mso-list: l1 level1 lfo1;">PlugIns</li> <ol style="margin-top: 0in;" start="1" type="a"> <li class="MsoListParagraphCxSpMiddle" style="margin-bottom: 8.0pt; margin-left: 0in; mso-add-space: auto; line-height: 105%; mso-list: l1 level2 lfo1;">Support pre/post build steps</li> <li class="MsoListParagraphCxSpMiddle" style="margin-bottom: 8.0pt; margin-left: 0in; mso-add-space: auto; line-height: 105%; mso-list: l1 level2 lfo1;">Support supplying functions</li> </ol> <li class="MsoListParagraphCxSpMiddle" style="margin-bottom: 8.0pt; margin-left: 0in; mso-add-space: auto; line-height: 105%; mso-list: l1 level1 lfo1;">UefiBuild - A wrapper around Edk2 build process that a Platform would subclass. <a href="https://github.com/microsoft/mu_pip_environment/blob/master/MuEnvironment/UefiBuild.py">https://github.com/microsoft/mu_pip_environment/blob/master/MuEnvironment/UefiBuild.py</a></li> <li class="MsoListParagraphCxSpLast" style="margin-bottom: 8.0pt; margin-left: 0in; mso-add-space: auto; line-height: 105%; mso-list: l1 level1 lfo1;">VarDict and ShellEnvironment - Manage the shell environment and all the name/value pairs used in the build process (including pre/post). </li> <li class="MsoListParagraphCxSpLast" style="margin-bottom: 8.0pt; margin-left: 0in; mso-add-space: auto; line-height: 105%; mso-list: l1 level1 lfo1;">Omnicache - Support a super cache of git repos to speed up creating and updating multiple work spaces and minimizing filesystem impact</li> </ol> Maintainers <ol style="margin-top: 0in;" start="1" type="1"> <li class="MsoListParagraphCxSpFirst" style="margin-bottom: 8.0pt; margin-left: 0in; mso-add-space: auto; line-height: 105%; mso-list: l8 level1 lfo2;">Sean Brogan</li> <li class="MsoListParagraphCxSpMiddle" style="margin-bottom: 8.0pt; margin-left: 0in; mso-add-space: auto; line-height: 105%; mso-list: l8 level1 lfo2;">Bret Barkelew</li> <li class="MsoListParagraphCxSpLast" style="margin-bottom: 8.0pt; margin-left: 0in; mso-add-space: auto; line-height: 105%; mso-list: l8 level1 lfo2;">Placeholder for existing maintainer from the basetools</li> </ol> License <ol style="margin-top: 0in;" start="1" type="1"> <li class="MsoListParagraph" style="margin-bottom: 8.0pt; margin-left: 0in; mso-add-space: auto; line-height: 105%; mso-list: l5 level1 lfo3;">BSD + Patent (edk2 aligned)</li> </ol> Contribution process and issue tracking <ol style="margin-top: 0in;" start="1" type="1"> <li class="MsoListParagraphCxSpFirst" style="margin-bottom: 8.0pt; margin-left: 0in; mso-add-space: auto; line-height: 105%; mso-list: l4 level1 lfo4;">Follow Github PR process for contributions and issue tracking</li> <ol style="margin-top: 0in;" start="1" type="a"> <li class="MsoListParagraphCxSpMiddle" style="margin-bottom: 8.0pt; margin-left: 0in; mso-add-space: auto; line-height: 105%; mso-list: l4 level2 lfo4;">Contributor forks repo in github</li> <li class="MsoListParagraphCxSpMiddle" style="margin-bottom: 8.0pt; margin-left: 0in; mso-add-space: auto; line-height: 105%; mso-list: l4 level2 lfo4;">Contributor creates branch for work</li> <li class="MsoListParagraphCxSpMiddle" style="margin-bottom: 8.0pt; margin-left: 0in; mso-add-space: auto; line-height: 105%; mso-list: l4 level2 lfo4;">Contributor updates release notes to indicate change (if necessary) </li> <li class="MsoListParagraphCxSpMiddle" style="margin-bottom: 8.0pt; margin-left: 0in; mso-add-space: auto; line-height: 105%; mso-list: l4 level2 lfo4;">Contributor submits PR to master branch of tianocore/Edk2-ToolEnv repo</li> <li class="MsoListParagraphCxSpMiddle" style="margin-bottom: 8.0pt; margin-left: 0in; mso-add-space: auto; line-height: 105%; mso-list: l4 level2 lfo4;">Review feedback is given in PR</li> <li class="MsoListParagraphCxSpMiddle" style="margin-bottom: 8.0pt; margin-left: 0in; mso-add-space: auto; line-height: 105%; mso-list: l4 level2 lfo4;">Python Tests are run on the repo (new contributions need unit tests)</li> <li class="MsoListParagraphCxSpMiddle" style="margin-bottom: 8.0pt; margin-left: 0in; mso-add-space: auto; line-height: 105%; mso-list: l4 level2 lfo4;">Python Style (flake8) must pass</li> <li class="MsoListParagraphCxSpLast" style="margin-bottom: 8.0pt; margin-left: 0in; mso-add-space: auto; line-height: 105%; mso-list: l4 level2 lfo4;">All review feedback must be completed, maintainers approved, and tests run successfully before PR is squash merged into master</li> </ol> </ol> Documentation <ol style="margin-top: 0in;" start="1" type="1"> <li class="MsoListParagraphCxSpFirst" style="margin-bottom: 8.0pt; margin-left: 0in; mso-add-space: auto; line-height: 105%; mso-list: l6 level1 lfo5;">Use Github IO documentation/wiki hosting</li> <ol style="margin-top: 0in;" start="1" type="a"> <li class="MsoListParagraphCxSpMiddle" style="margin-bottom: 8.0pt; margin-left: 0in; mso-add-space: auto; line-height: 105%; mso-list: l6 level2 lfo5;">Example content</li> </ol> </ol>  i.  <a href="https://microsoft.github.io/mu/dyn/mu_pip_environment/developing/">https://microsoft.github.io/mu/dyn/mu_pip_environment/developing/</a>  ii. <a href="https://microsoft.github.io/mu/dyn/mu_pip_environment/publishing/">https://microsoft.github.io/mu/dyn/mu_pip_environment/publishing/</a> <ol style="margin-top: 0in;" start="2" type="1"> <li class="MsoListParagraphCxSpMiddle" style="margin-bottom: 8.0pt; margin-left: 0in; mso-add-space: auto; line-height: 105%; mso-list: l6 level1 lfo5;">Readme at root of repo</li> <ol style="margin-top: 0in;" start="1" type="a"> <li class="MsoListParagraphCxSpLast" style="margin-bottom: 8.0pt; margin-left: 0in; mso-add-space: auto; line-height: 105%; mso-list: l6 level2 lfo5;">Example: <a href="https://github.com/Microsoft/mu_pip_environment">https://github.com/Microsoft/mu_pip_environment</a></li> </ol> </ol> CI Builds <ol style="margin-top: 0in;" start="1" type="1"> <li class="MsoListParagraphCxSpFirst" style="margin-bottom: 8.0pt; margin-left: 0in; mso-add-space: auto; line-height: 105%; mso-list: l9 level1 lfo6;">CI build process using dev ops</li> <li class="MsoListParagraphCxSpMiddle" style="margin-bottom: 8.0pt; margin-left: 0in; mso-add-space: auto; line-height: 105%; mso-list: l9 level1 lfo6;">Validation is done thru build process</li> <li class="MsoListParagraphCxSpMiddle" style="margin-bottom: 8.0pt; margin-left: 0in; mso-add-space: auto; line-height: 105%; mso-list: l9 level1 lfo6;">Release publication done thru manual CI Build</li> <li class="MsoListParagraphCxSpMiddle" style="margin-bottom: 8.0pt; margin-left: 0in; mso-add-space: auto; line-height: 105%; mso-list: l9 level1 lfo6;">Examples from Mu-Environment</li> <ol style="margin-top: 0in;" start="1" type="a"> <li class="MsoListParagraphCxSpMiddle" style="margin-bottom: 8.0pt; margin-left: 0in; mso-add-space: auto; line-height: 105%; mso-list: l9 level2 lfo6;">Windows CI - <a href="https://dev.azure.com/projectmu/mu%20pip/_build?definitionId=10">https://dev.azure.com/projectmu/mu%20pip/_build?definitionId=10</a></li> <li class="MsoListParagraphCxSpMiddle" style="margin-bottom: 8.0pt; margin-left: 0in; mso-add-space: auto; line-height: 105%; mso-list: l9 level2 lfo6;">Linux CI - <a href="https://dev.azure.com/projectmu/mu%20pip/_build?definitionId=11">https://dev.azure.com/projectmu/mu%20pip/_build?definitionId=11</a></li> <li class="MsoListParagraphCxSpLast" style="margin-bottom: 8.0pt; margin-left: 0in; mso-add-space: auto; line-height: 105%; mso-list: l9 level2 lfo6;">Publishing - <a href="https://dev.azure.com/projectmu/mu%20pip/_build?definitionId=17">https://dev.azure.com/projectmu/mu%20pip/_build?definitionId=17</a></li> </ol> </ol> Release <ol style="margin-top: 0in;" start="1" type="1"> <li class="MsoListParagraphCxSpFirst" style="margin-bottom: 8.0pt; margin-left: 0in; mso-add-space: auto; line-height: 105%; mso-list: l2 level1 lfo7;">Release to Pypi as Edk2-ToolEnv for easy usage in product environment</li> <ol style="margin-top: 0in;" start="1" type="a"> <li class="MsoListParagraphCxSpMiddle" style="margin-bottom: 8.0pt; margin-left: 0in; mso-add-space: auto; line-height: 105%; mso-list: l2 level2 lfo7;">Versioned follows: Aa.bb.cc.dd</li> <li class="MsoListParagraphCxSpMiddle" style="margin-bottom: 8.0pt; margin-left: 0in; mso-add-space: auto; line-height: 105%; mso-list: l2 level2 lfo7;">AA == Major version. Changes don’t need to be backward compatible</li> <li class="MsoListParagraphCxSpMiddle" style="margin-bottom: 8.0pt; margin-left: 0in; mso-add-space: auto; line-height: 105%; mso-list: l2 level2 lfo7;">BB == Minor version. Significant new features. Backward compatibility maintained </li> <li class="MsoListParagraphCxSpMiddle" style="margin-bottom: 8.0pt; margin-left: 0in; mso-add-space: auto; line-height: 105%; mso-list: l2 level2 lfo7;">CC == Bug fix/patch/small optional feature</li> <li class="MsoListParagraphCxSpMiddle" style="margin-bottom: 8.0pt; margin-left: 0in; mso-add-space: auto; line-height: 105%; mso-list: l2 level2 lfo7;">DD == build/Release version. </li> <li class="MsoListParagraphCxSpMiddle" style="margin-bottom: 8.0pt; margin-left: 0in; mso-add-space: auto; line-height: 105%; mso-list: l2 level2 lfo7;">Package on Pypi will be owned by Tianocore group</li> </ol> <li class="MsoListParagraphCxSpLast" style="margin-bottom: 8.0pt; margin-left: 0in; mso-add-space: auto; line-height: 105%; mso-list: l2 level1 lfo7;">Example for mu-environment: <a href="https://pypi.org/project/mu-environment/">https://pypi.org/project/mu-environment/</a></li> </ol> Other Notes <ol style="margin-top: 0in;" start="1" type="1"> <li class="MsoListParagraphCxSpFirst" style="margin-bottom: 8.0pt; margin-left: 0in; mso-add-space: auto; line-height: 105%; mso-list: l7 level1 lfo8;">Only support Python 3 (prefer 3.7+)</li> <li class="MsoListParagraphCxSpMiddle" style="margin-bottom: 8.0pt; margin-left: 0in; mso-add-space: auto; line-height: 105%; mso-list: l7 level1 lfo8;"><span style="mso-ascii-font-family: Calibri; mso-ascii-theme-font: minor-latin; mso-hansi-font-family: Calibri; mso-hansi-theme-font: minor-latin; mso-bidi-font-family: 'Times New Roman'; mso-bidi-theme-font: minor-bidi;">This was discussed on the edk2 design meetings (4/18) <a href="https://edk2.groups.io/g/devel/files/Designs/2019/0418">https://edk2.groups.io/g/devel/files/Designs/2019/0418</a><span style="mso-ascii-font-family: Calibri; mso-ascii-theme-font: minor-latin; mso-hansi-font-family: Calibri; mso-hansi-theme-font: minor-latin; mso-bidi-font-family: 'Times New Roman'; mso-bidi-theme-font: minor-bidi;"></li> <li class="MsoListParagraphCxSpMiddle" style="margin-bottom: 8.0pt; margin-left: 0in; mso-add-space: auto; line-height: 105%; mso-list: l7 level1 lfo8;"><span style="mso-ascii-font-family: Calibri; mso-ascii-theme-font: minor-latin; mso-hansi-font-family: Calibri; mso-hansi-theme-font: minor-latin; mso-bidi-font-family: 'Times New Roman'; mso-bidi-theme-font: minor-bidi;">There is RFC #2. The 1st RFC is for Edk2-Library which this RFC depends on. </li> <li class="MsoListParagraphCxSpMiddle" style="margin-bottom: 8.0pt; margin-left: 0in; mso-add-space: auto; line-height: 105%; mso-list: l7 level1 lfo8;"><span style="mso-ascii-font-family: Calibri; mso-ascii-theme-font: minor-latin; mso-hansi-font-family: Calibri; mso-hansi-theme-font: minor-latin; mso-bidi-font-family: 'Times New Roman'; mso-bidi-theme-font: minor-bidi;">Simple example of usage on an open platform: <a href="https://github.com/ms-iot/MU_PLATFORM_NXP/blob/master/NXP/MCIMX8M_MINI_EVK_2GB/PlatformBuildWorker.py">https://github.com/ms-iot/MU_PLATFORM_NXP/blob/master/NXP/MCIMX8M_MINI_EVK_2GB/PlatformBuildWorker.py</a><span style="mso-ascii-font-family: Calibri; mso-ascii-theme-font: minor-latin; mso-hansi-font-family: Calibri; mso-hansi-theme-font: minor-latin; mso-bidi-font-family: 'Times New Roman'; mso-bidi-theme-font: minor-bidi;"></li> <li class="MsoListParagraphCxSpMiddle" style="margin-bottom: 8.0pt; margin-left: 0in; mso-add-space: auto; line-height: 105%; mso-list: l7 level1 lfo8;">Example of CI of something like Tianocore: <span style="mso-ascii-font-family: Calibri; mso-ascii-theme-font: minor-latin; mso-hansi-font-family: Calibri; mso-hansi-theme-font: minor-latin; mso-bidi-font-family: 'Times New Roman'; mso-bidi-theme-font: minor-bidi;"></li> <ol style="margin-top: 0in;" start="1" type="a"> <li class="MsoListParagraphCxSpMiddle" style="margin-bottom: 8.0pt; margin-left: 0in; mso-add-space: auto; line-height: 105%; mso-list: l7 level2 lfo8;">Windows - Mu_Basecore: <a href="https://dev.azure.com/projectmu/mu/_build?definitionId=4&view=buildsHistory">https://dev.azure.com/projectmu/mu/_build?definitionId=4&view=buildsHistory</a><span style="mso-ascii-font-family: Calibri; mso-ascii-theme-font: minor-latin; mso-hansi-font-family: Calibri; mso-hansi-theme-font: minor-latin; mso-bidi-font-family: 'Times New Roman'; mso-bidi-theme-font: minor-bidi;"></li> <li class="MsoListParagraphCxSpMiddle" style="margin-bottom: 8.0pt; margin-left: 0in; mso-add-space: auto; line-height: 105%; mso-list: l7 level2 lfo8;">Linux - Mu_Basecore: <a href="https://dev.azure.com/projectmu/mu/_build?definitionId=19&view=buildsHistory">https://dev.azure.com/projectmu/mu/_build?definitionId=19&view=buildsHistory</a><span style="mso-ascii-font-family: Calibri; mso-ascii-theme-font: minor-latin; mso-hansi-font-family: Calibri; mso-hansi-theme-font: minor-latin; mso-bidi-font-family: 'Times New Roman'; mso-bidi-theme-font: minor-bidi;"></li> <li class="MsoListParagraphCxSpLast" style="margin-bottom: 8.0pt; margin-left: 0in; mso-add-space: auto; line-height: 105%; mso-list: l7 level2 lfo8;">Windows - Project Mu Plus code: <a href="https://dev.azure.com/projectmu/mu/_build?definitionId=6&view=buildsHistory">https://dev.azure.com/projectmu/mu/_build?definitionId=6&view=buildsHistory</a> <span style="mso-ascii-font-family: Calibri; mso-ascii-theme-font: minor-latin; mso-hansi-font-family: Calibri; mso-hansi-theme-font: minor-latin; mso-bidi-font-family: 'Times New Roman'; mso-bidi-theme-font: minor-bidi;"></li> </ol> </ol> <span style="mso-ascii-font-family: Calibri; mso-ascii-theme-font: minor-latin; mso-hansi-font-family: Calibri; mso-hansi-theme-font: minor-latin; mso-bidi-font-family: 'Times New Roman'; mso-bidi-theme-font: minor-bidi;"> Timeline <ol style="margin-top: 0in;" start="1" type="1"> <li class="MsoListParagraph" style="margin-bottom: 8.0pt; margin-left: 0in; mso-add-space: auto; line-height: 105%; mso-list: l3 level1 lfo9;">RFC open for comment thru 5/21/2019 since the same as RFC Edk2-Library</li> </ol> <div width="1" style="color:white;clear:both">_._,_._,_</div> <hr> Groups.io Links: You receive all messages sent to this group. <a target="_blank" href="https://edk2.groups.io/g/devel/message/40567">View/Reply Online (#40567)</a> | | <a target="_blank" href="https://groups.io/mt/31614611/1813853">Mute This Topic</a> | <a href="https://edk2.groups.io/g/devel/post">New Topic</a> <a href="https://edk2.groups.io/g/devel/editsub/1813853">Your Subscription</a> | <a href="mailto:devel+owner@edk2.groups.io">Contact Group Owner</a> | <a href="https://edk2.groups.io/g/devel/unsub">Unsubscribe</a> [edk2-devel-archive@redhat.com] <div width="1" style="color:white;clear:both">_._,_._,_</div>