[edk2-devel] [PATCH] BaseTools:Extend the binary cache to support library cache

[Christian Rodriguez]

Hi Steven,

After testing the bug reported, it seems that the dependency problem isn't the root cause. We don't need to change the way it skips having to rebuild libraries. The cache is missing the library build artifacts as you mentioned before and this is the root cause of the rebuild failure. This patch should fix that, but you can simplify it by not adding a new tracking system and keeping the dependency reduction as before. That will save the performance because it doesn't have to do extra checking.

Please see attached patch for minimal change needed.

Thanks,
Christian

>-----Original Message-----
>From: devel at edk2.groups.io [mailto:devel at edk2.groups.io] On Behalf Of
>Steven Shi
>Sent: Tuesday, May 28, 2019 1:52 AM
>To: devel at edk2.groups.io
>Cc: Gao, Liming <liming.gao at intel.com>; Feng, Bob C
><bob.c.feng at intel.com>; Rodriguez, Christian
><christian.rodriguez at intel.com>; Fan, ZhijuX <zhijux.fan at intel.com>
>Subject: [edk2-devel] [PATCH] BaseTools:Extend the binary cache to support
>library cache
>
>https://bugzilla.tianocore.org/show_bug.cgi?id=1797
>
>Current binary cache doesn't support to save and restore
>the library module. If a driver module cache miss happen,
>all its dependency library modules need rebuild which
>is very time-consuming. This patch is to entend the binary
>cache to support library.
>
>Cc: Liming Gao <liming.gao at intel.com>
>Cc: Bob Feng <bob.c.feng at intel.com>
>Cc: Christian Rodriguez <christian.rodriguez at intel.com>
>Signed-off-by: Steven Shi <steven.shi at intel.com>
>---
> BaseTools/Source/Python/AutoGen/AutoGen.py | 17 ++++++++++++++---
> BaseTools/Source/Python/build/build.py     |  3 ++-
> 2 files changed, 16 insertions(+), 4 deletions(-)
>
>diff --git a/BaseTools/Source/Python/AutoGen/AutoGen.py
>b/BaseTools/Source/Python/AutoGen/AutoGen.py
>index a5bef4f7c6..aeb63f52c5 100644
>--- a/BaseTools/Source/Python/AutoGen/AutoGen.py
>+++ b/BaseTools/Source/Python/AutoGen/AutoGen.py
>@@ -2578,7 +2578,7 @@ class ModuleAutoGen(AutoGen):
>         self.AutoGenDepSet = set()
>         self.ReferenceModules = []
>         self.ConstPcd                  = {}
>-
>+        self.CacheRestored             = False
>
>     def __repr__(self):
>         return "%s [%s]" % (self.MetaFile, self.Arch)
>@@ -3906,6 +3906,12 @@ class ModuleAutoGen(AutoGen):
>             ModuleFile = path.join(self.OutputDir, self.Name + '.inf')
>             if os.path.exists(ModuleFile):
>                 shutil.copy2(ModuleFile, FileDir)
>+        else:
>+            OutputDir = self.OutputDir.replace('\\', '/').strip('/')
>+            DebugDir = self.DebugDir.replace('\\', '/').strip('/')
>+            for Item in self.CodaTargetList:
>+                File = Item.Target.Path.replace('\\', '/').strip('/').replace(DebugDir,
>'').replace(OutputDir, '').strip('/')
>+                self.OutputFile.add(File)
>         if not self.OutputFile:
>             Ma = self.BuildDatabase[self.MetaFile, self.Arch, self.BuildTarget,
>self.ToolChain]
>             self.OutputFile = Ma.Binaries
>@@ -3949,6 +3955,7 @@ class ModuleAutoGen(AutoGen):
>                                 destination_dir = os.path.dirname(destination_file)
>                                 CreateDirectory(destination_dir)
>                                 shutil.copy2(File, destination_dir)
>+                    self.CacheRestored = True
>                     if self.Name == "PcdPeim" or self.Name == "PcdDxe":
>                         CreatePcdDatabaseCode(self, TemplateString(),
>TemplateString())
>                     return True
>@@ -3987,7 +3994,9 @@ class ModuleAutoGen(AutoGen):
>         self.GenFfsList = GenFfsList
>         if not self.IsLibrary and CreateLibraryMakeFile:
>             for LibraryAutoGen in self.LibraryAutoGenList:
>-                LibraryAutoGen.CreateMakeFile()
>+                # Only create makefile for libraries which have not been restored
>+                if not LibraryAutoGen.CacheRestored:
>+                    LibraryAutoGen.CreateMakeFile()
>
>         if self.CanSkip():
>             return
>@@ -4030,7 +4039,9 @@ class ModuleAutoGen(AutoGen):
>
>         if not self.IsLibrary and CreateLibraryCodeFile:
>             for LibraryAutoGen in self.LibraryAutoGenList:
>-                LibraryAutoGen.CreateCodeFile()
>+                # Only create autogen code for libraries which have not been
>restored
>+                if not LibraryAutoGen.CacheRestored:
>+                    LibraryAutoGen.CreateCodeFile()
>
>         if self.CanSkip():
>             return
>diff --git a/BaseTools/Source/Python/build/build.py
>b/BaseTools/Source/Python/build/build.py
>index 80ceb98310..f1f4c07980 100644
>--- a/BaseTools/Source/Python/build/build.py
>+++ b/BaseTools/Source/Python/build/build.py
>@@ -341,7 +341,8 @@ class ModuleMakeUnit(BuildUnit):
>     #   @param  Target      The build target name, one of gSupportedTarget
>     #
>     def __init__(self, Obj, Target):
>-        Dependency = [ModuleMakeUnit(La, Target) for La in
>Obj.LibraryAutoGenList]
>+        # Skip the dependency modules which are already restored from cache
>+        Dependency = [ModuleMakeUnit(La, Target) for La in
>Obj.LibraryAutoGenList if not La.CacheRestored]
>         BuildUnit.__init__(self, Obj, Obj.BuildCommand, Target, Dependency,
>Obj.MakeFileDir)
>         if Target in [None, "", "all"]:
>             self.Target = "tbuild"
>--
>2.17.1.windows.2
>
>
>


-=-=-=-=-=-=-=-=-=-=-=-
Groups.io Links: You receive all messages sent to this group.

View/Reply Online (#41513): https://edk2.groups.io/g/devel/message/41513
Mute This Topic: https://groups.io/mt/31819658/1813853
Group Owner: devel+owner at edk2.groups.io
Unsubscribe: https://edk2.groups.io/g/devel/unsub  [edk2-devel-archive at redhat.com]
-=-=-=-=-=-=-=-=-=-=-=-

-------------- next part --------------
A non-text attachment was scrubbed...
Name: 0001-Add-library-copy-to-cache.patch
Type: application/octet-stream
Size: 1335 bytes
Desc: 0001-Add-library-copy-to-cache.patch
URL: <http://listman.redhat.com/archives/edk2-devel-archive/attachments/20190528/d9f7921d/attachment.obj>