[edk2-devel] [Patch 00/11 V4] Enable multiple process AutoGen

[Bob Feng]

BZ: https://bugzilla.tianocore.org/show_bug.cgi?id=1875

In order to improve the build performance, we implemented
multiple-processes AutoGen. This change will reduce 20% time
for AutoGen phase.

The design document can be got from:
https://edk2.groups.io/g/devel/files/Designs/2019/0627/Multiple-thread-AutoGen.pdf

This patch serial pass the build of Ovmf, MinKabylake, MinPurley, packages
under Edk2 repository and intel client and server platforms.

V4:
Add one more patch 11/11 to enhance this feature. 1-10 are the same as V3
1. Set Log queue maxsize as thread number * 10
2. enhance ModuleUniqueBaseName function
3. fix bugs of build option pcd in sub Process
4. enhance error handling. Handle the exception of
KeyboardInterrup and exceptions happen in subprocess.
5. fix the issue of shared fixed pcd between module and lib.
6. fix bug in the function of duplicate modules handling.
V3:
1. Fixed incremental build issue.
2. Set AutoGen worker number to be align with "-n THREADNUMBER"
3. Enable block log queue.
V2: 
1. The first version missed autogen related commit 
from e812a812c1a0800c49e11507cb46222351520cc7. V2 add those commit
back.
2. Move CreateAsBuildInf into AutoGenWorker process
3. Save GlobalVar_<platform guid>_<arch>.bin to build folder.
4. Regenerate patches based on master bb824f685d
Feng, Bob C (11):
  BaseTools: Singleton the object to handle build conf file
  BaseTools: Split WorkspaceAutoGen._InitWorker into multiple functions
  BaseTools: Add functions to get platform scope build options
  BaseTools: Decouple AutoGen Objects
  BaseTools: Enable Multiple Process AutoGen
  BaseTools: Add shared data for processes
  BaseTools: Add LogAgent to support multiple process Autogen
  BaseTools: Move BuildOption parser out of build.py
  BaseTools: Add the support for python 2
  BaseTools: Enable block queue log agent.
  BaseTools: Enhance Multiple-Process AutoGen

 BaseTools/Source/Python/AutoGen/AutoGen.py    | 4227 +----------------
 .../Source/Python/AutoGen/AutoGenWorker.py    |  257 +
 .../Source/Python/AutoGen/BuildEngine.py      |   22 +
 BaseTools/Source/Python/AutoGen/DataPipe.py   |  160 +
 BaseTools/Source/Python/AutoGen/GenC.py       |    6 +-
 .../Source/Python/AutoGen/ModuleAutoGen.py    | 1903 ++++++++
 .../Python/AutoGen/ModuleAutoGenHelper.py     |  619 +++
 .../Source/Python/AutoGen/PlatformAutoGen.py  | 1512 ++++++
 .../Source/Python/AutoGen/WorkspaceAutoGen.py |  907 ++++
 BaseTools/Source/Python/Common/EdkLogger.py   |  119 +-
 BaseTools/Source/Python/Common/Misc.py        |    1 -
 .../Python/Common/TargetTxtClassObject.py     |   28 +-
 .../Python/Common/ToolDefClassObject.py       |    6 +-
 BaseTools/Source/Python/GenFds/GenFds.py      |    4 +-
 .../Python/GenFds/GenFdsGlobalVariable.py     |   54 +-
 .../Python/PatchPcdValue/PatchPcdValue.py     |    1 -
 .../Source/Python/Workspace/DscBuildData.py   |   38 +-
 .../Source/Python/Workspace/InfBuildData.py   |   39 +
 .../Python/Workspace/WorkspaceCommon.py       |    4 +
 .../Python/Workspace/WorkspaceDatabase.py     |    3 +
 BaseTools/Source/Python/build/BuildReport.py  |    4 +-
 BaseTools/Source/Python/build/build.py        |  370 +-
 BaseTools/Source/Python/build/buildoptions.py |   92 +
 23 files changed, 5921 insertions(+), 4455 deletions(-)
 create mode 100644 BaseTools/Source/Python/AutoGen/AutoGenWorker.py
 create mode 100644 BaseTools/Source/Python/AutoGen/DataPipe.py
 create mode 100644 BaseTools/Source/Python/AutoGen/ModuleAutoGen.py
 create mode 100644 BaseTools/Source/Python/AutoGen/ModuleAutoGenHelper.py
 create mode 100644 BaseTools/Source/Python/AutoGen/PlatformAutoGen.py
 create mode 100644 BaseTools/Source/Python/AutoGen/WorkspaceAutoGen.py
 create mode 100644 BaseTools/Source/Python/build/buildoptions.py

-- 
2.20.1.windows.1


-=-=-=-=-=-=-=-=-=-=-=-
Groups.io Links: You receive all messages sent to this group.

View/Reply Online (#44491): https://edk2.groups.io/g/devel/message/44491
Mute This Topic: https://groups.io/mt/32640226/1813853
Group Owner: devel+owner at edk2.groups.io
Unsubscribe: https://edk2.groups.io/g/devel/unsub  [edk2-devel-archive at redhat.com]
-=-=-=-=-=-=-=-=-=-=-=-