[EXTERNAL] Re: [edk2-devel] Do the edk2 CI unit tests work in Linux?

Steven Shi steven.shi at intel.com
Fri Jul 2 01:36:31 UTC 2021 

I already installed mono, but NuGet.exe still fails to work in my side. I'm behind a proxy but don't know how to configure the NuGet.exe proxy in .pytool. Below is my verbose error message.

(unit_test_env) jshi19 at ub2-uefi-b01:~/wksp_efi/edk2-2$ stuart_update -c .pytool/CISettings.py TOOL_CHAIN_TAG=GCC5 -v
... ...
SECTION - Start Invocable Tool
SECTION - Initial update of environment
DEBUG - --- self_describing_environment.update_extdeps()
DEBUG - Verify 'mu_nasm' returning 'False'.
DEBUG - Verify 'edk2-acpica-iasl' returning 'False'.
DEBUG - Creating 5 threads for the SDE update
UpdatingDEBUG - Verify 'gcc_aarch64_linux' returning 'True'.
DEBUG - Verify 'gcc_riscv64_unknown' returning 'True'.
DEBUG - Verify 'mu_nasm' returning 'False'.
DEBUG - Verify 'gcc_arm_linux' returning 'True'.
DEBUG - Verify 'edk2-acpica-iasl' returning 'False'.
DEBUG - Cleaning dependency directory for 'mu_nasm'...
DEBUG - Cleaning dependency directory for 'edk2-acpica-iasl'...
INFO - Cmd to run is: mono /home/jshi19/wksp_efi/unit_test_env/lib/python3.8/site-packages/edk2toolext/bin/NuGet.exe locals global-packages -list
INFO - Cmd to run is: mono /home/jshi19/wksp_efi/unit_test_env/lib/python3.8/site-packages/edk2toolext/bin/NuGet.exe locals global-packages -list
INFO - ------------------------------------------------
INFO - ------------------------------------------------
INFO - --------------Cmd Output Starting---------------
INFO - --------------Cmd Output Starting---------------
INFO - ------------------------------------------------
INFO - ------------------------------------------------
...INFO - Could not load file or assembly 'System.Net.Http, Version=4.0.0.0, Culture=neutral, PublicKeyToken=b03f5f7f11d50a3a' or one of its dependencies.
INFO -   Could not load file or assembly 'System.Net.Http, Version=4.0.0.0, Culture=neutral, PublicKeyToken=b03f5f7f11d50a3a' or one of its dependencies.
INFO -   Could not load file or assembly 'System.Net.Http, Version=4.0.0.0, Culture=neutral, PublicKeyToken=b03f5f7f11d50a3a' or one of its dependencies.
INFO -   Could not load file or assembly 'System.Net.Http, Version=4.0.0.0, Culture=neutral, PublicKeyToken=b03f5f7f11d50a3a' or one of its dependencies.
INFO -   Could not load file or assembly 'System.Net.Http, Version=4.0.0.0, Culture=neutral, PublicKeyToken=b03f5f7f11d50a3a' or one of its dependencies.
INFO - System.AggregateException: One or more errors occurred. (Could not load file or assembly 'System.Net.Http, Version=4.0.0.0, Culture=neutral, PublicKeyToken=b03f5f7f11d50a3a' or one of its dependencies.) (Could not load file or assembly 'System.Net.Http, Version=4.0.0.0, Culture=neutral, PublicKeyToken=b03f5f7f11d50a3a' or one of its dependencies.) (Could not load file or assembly 'System.Net.Http, Version=4.0.0.0, Culture=neutral, PublicKeyToken=b03f5f7f11d50a3a' or one of its dependencies.) (Could not load file or assembly 'System.Net.Http, Version=4.0.0.0, Culture=neutral, PublicKeyToken=b03f5f7f11d50a3a' or one of its dependencies.) (Could not load file or assembly 'System.Net.Http, Version=4.0.0.0, Culture=neutral, PublicKeyToken=b03f5f7f11d50a3a' or one of its dependencies.) ---> System.IO.FileNotFoundException: Could not load file or assembly 'System.Net.Http, Version=4.0.0.0, Culture=neutral, PublicKeyToken=b03f5f7f11d50a3a' or one of its dependencies.
INFO -    --- End of inner exception stack trace ---
... ...


If I run the Cmd manually, I get below error:
(unit_test_env) jshi19 at ub2-uefi-b01:~/wksp_efi/edk2-2$ mono /home/jshi19/wksp_efi/unit_test_env/lib/python3.8/site-packages/edk2toolext/bin/NuGet.exe locals global-packages -list
Could not load file or assembly 'System.Net.Http, Version=4.0.0.0, Culture=neutral, PublicKeyToken=b03f5f7f11d50a3a' or one of its dependencies.
  Could not load file or assembly 'System.Net.Http, Version=4.0.0.0, Culture=neutral, PublicKeyToken=b03f5f7f11d50a3a' or one of its dependencies.
  Could not load file or assembly 'System.Net.Http, Version=4.0.0.0, Culture=neutral, PublicKeyToken=b03f5f7f11d50a3a' or one of its dependencies.
  Could not load file or assembly 'System.Net.Http, Version=4.0.0.0, Culture=neutral, PublicKeyToken=b03f5f7f11d50a3a' or one of its dependencies.
  Could not load file or assembly 'System.Net.Http, Version=4.0.0.0, Culture=neutral, PublicKeyToken=b03f5f7f11d50a3a' or one of its dependencies.
System.AggregateException: One or more errors occurred. (Could not load file or assembly 'System.Net.Http, Version=4.0.0.0, Culture=neutral, PublicKeyToken=b03f5f7f11d50a3a' or one of its dependencies.) (Could not load file or assembly 'System.Net.Http, Version=4.0.0.0, Culture=neutral, PublicKeyToken=b03f5f7f11d50a3a' or one of its dependencies.) (Could not load file or assembly 'System.Net.Http, Version=4.0.0.0, Culture=neutral, PublicKeyToken=b03f5f7f11d50a3a' or one of its dependencies.) (Could not load file or assembly 'System.Net.Http, Version=4.0.0.0, Culture=neutral, PublicKeyToken=b03f5f7f11d50a3a' or one of its dependencies.) (Could not load file or assembly 'System.Net.Http, Version=4.0.0.0, Culture=neutral, PublicKeyToken=b03f5f7f11d50a3a' or one of its dependencies.) ---> System.IO.FileNotFoundException: Could not load file or assembly 'System.Net.Http, Version=4.0.0.0, Culture=neutral, PublicKeyToken=b03f5f7f11d50a3a' or one of its dependencies.
   --- End of inner exception stack trace ---
  at NuGet.CommandLine.Program.Initialize (NuGet.IFileSystem fileSystem, NuGet.CommandLine.IConsole console) [0x000ce] in <ff502667b3fe4a85827635c6fe6b0d0d>:0
  at NuGet.CommandLine.Program.MainCore (System.String workingDirectory, System.String[] args) [0x000d8] in <ff502667b3fe4a85827635c6fe6b0d0d>:0
---> (Inner Exception #0) System.IO.FileNotFoundException: Could not load file or assembly 'System.Net.Http, Version=4.0.0.0, Culture=neutral, PublicKeyToken=b03f5f7f11d50a3a' or one of its dependencies.
File name: 'System.Net.Http, Version=4.0.0.0, Culture=neutral, PublicKeyToken=b03f5f7f11d50a3a'<---

---> (Inner Exception #1) System.IO.FileNotFoundException: Could not load file or assembly 'System.Net.Http, Version=4.0.0.0, Culture=neutral, PublicKeyToken=b03f5f7f11d50a3a' or one of its dependencies.
File name: 'System.Net.Http, Version=4.0.0.0, Culture=neutral, PublicKeyToken=b03f5f7f11d50a3a'<---

---> (Inner Exception #2) System.IO.FileNotFoundException: Could not load file or assembly 'System.Net.Http, Version=4.0.0.0, Culture=neutral, PublicKeyToken=b03f5f7f11d50a3a' or one of its dependencies.
File name: 'System.Net.Http, Version=4.0.0.0, Culture=neutral, PublicKeyToken=b03f5f7f11d50a3a'<---

---> (Inner Exception #3) System.IO.FileNotFoundException: Could not load file or assembly 'System.Net.Http, Version=4.0.0.0, Culture=neutral, PublicKeyToken=b03f5f7f11d50a3a' or one of its dependencies.
File name: 'System.Net.Http, Version=4.0.0.0, Culture=neutral, PublicKeyToken=b03f5f7f11d50a3a'<---

---> (Inner Exception #4) System.IO.FileNotFoundException: Could not load file or assembly 'System.Net.Http, Version=4.0.0.0, Culture=neutral, PublicKeyToken=b03f5f7f11d50a3a' or one of its dependencies.
File name: 'System.Net.Http, Version=4.0.0.0, Culture=neutral, PublicKeyToken=b03f5f7f11d50a3a'<---


Thanks

Steven Shi


From: Bret Barkelew <Bret.Barkelew at microsoft.com>
Sent: Friday, July 2, 2021 9:04 AM
To: Shi, Steven <steven.shi at intel.com>; devel at edk2.groups.io; rebecca at nuviainc.com
Subject: RE: [EXTERNAL] Re: [edk2-devel] Do the edk2 CI unit tests work in Linux?

I believe the Azure DevOps build agents that are used by the EDK2 CI pipeline already have mono installed.

- Bret

From: Shi, Steven<mailto:steven.shi at intel.com>
Sent: Thursday, July 1, 2021 5:54 PM
To: devel at edk2.groups.io<mailto:devel at edk2.groups.io>; Bret Barkelew<mailto:Bret.Barkelew at microsoft.com>; rebecca at nuviainc.com<mailto:rebecca at nuviainc.com>
Subject: RE: [EXTERNAL] Re: [edk2-devel] Do the edk2 CI unit tests work in Linux?

Hi Bret, Rebecca,
Thanks for the info. Does edk2 CI run the .pytool unit tests in Linux? If yes, how does edk2 CI solve the nuget dependency problem in Linux right now?


Thanks

Steven Shi


From: devel at edk2.groups.io<mailto:devel at edk2.groups.io> <devel at edk2.groups.io<mailto:devel at edk2.groups.io>> On Behalf Of Bret Barkelew via groups.io
Sent: Wednesday, June 30, 2021 1:02 AM
To: devel at edk2.groups.io<mailto:devel at edk2.groups.io>; rebecca at nuviainc.com<mailto:rebecca at nuviainc.com>; Shi, Steven <steven.shi at intel.com<mailto:steven.shi at intel.com>>
Subject: Re: [EXTERNAL] Re: [edk2-devel] Do the edk2 CI unit tests work in Linux?

Hmmm. If that works well, we might be able to investigate carrying both and picking based on the host OS.

- Bret

From: devel at edk2.groups.io<mailto:devel at edk2.groups.io> <devel at edk2.groups.io<mailto:devel at edk2.groups.io>> on behalf of Rebecca Cran via groups.io <rebecca=nuviainc.com at groups.io<mailto:rebecca=nuviainc.com at groups.io>>
Sent: Tuesday, June 29, 2021 9:42:41 AM
To: devel at edk2.groups.io<mailto:devel at edk2.groups.io> <devel at edk2.groups.io<mailto:devel at edk2.groups.io>>; steven.shi at intel.com<mailto:steven.shi at intel.com> <steven.shi at intel.com<mailto:steven.shi at intel.com>>
Subject: [EXTERNAL] Re: [edk2-devel] Do the edk2 CI unit tests work in Linux?


Your Linux distribution should have a nuget package available (e.g. https://packages.ubuntu.com/hirsute/nuget<https://nam06.safelinks.protection.outlook.com/?url=https%3A%2F%2Fpackages.ubuntu.com%2Fhirsute%2Fnuget&data=04%7C01%7Cbret.barkelew%40microsoft.com%7C28bf7b813a9a4d6e365508d93cf3fd8a%7C72f988bf86f141af91ab2d7cd011db47%7C1%7C0%7C637607840913328150%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C1000&sdata=Z7vCN9JUmx1kQks%2FuWKjcnMRo7koWoyxOET0bnSbN6U%3D&reserved=0>) but the EDK2 CI is designed to use the copy of nuget installed with the edk2toolext package.



--
Rebecca Cran


On 6/29/21 9:44 AM, Steven Shi wrote:

Cran, Thanks. I'm curious whether the .pytool has native Linux executable binary to run? It looks the edk2 CI framework has more dependency in Linux than Windows.





Thanks



Steven Shi

Intel\IAGS\SFP\FIA (Firmware Infrastructure Automation)





From: devel at edk2.groups.io<mailto:devel at edk2.groups.io> <devel at edk2.groups.io><mailto:devel at edk2.groups.io> On Behalf Of Rebecca Cran
Sent: Tuesday, June 29, 2021 8:13 PM
To: devel at edk2.groups.io<mailto:devel at edk2.groups.io>; Shi, Steven <steven.shi at intel.com><mailto:steven.shi at intel.com>
Subject: Re: [edk2-devel] Do the edk2 CI unit tests work in Linux?



NuGet.exe is a .NET assembly/executable, so on Linux you'll need to install Mono (https://www.mono-project.com/<https://nam06.safelinks.protection.outlook.com/?url=https%3A%2F%2Fwww.mono-project.com%2F&data=04%7C01%7Cbret.barkelew%40microsoft.com%7C28bf7b813a9a4d6e365508d93cf3fd8a%7C72f988bf86f141af91ab2d7cd011db47%7C1%7C0%7C637607840913338106%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C1000&sdata=uPkpInnWMx%2FP35GUXlbffQwxbLya9DDDegyLJvtuScU%3D&reserved=0>).



--

Rebecca Cran





On 6/28/21 7:45 PM, Steven Shi wrote:

Hello,

I'm interested in the edk2 CI unit tests(https://github.com/tianocore/edk2/tree/master/.pytool<https://nam06.safelinks.protection.outlook.com/?url=https%3A%2F%2Fgithub.com%2Ftianocore%2Fedk2%2Ftree%2Fmaster%2F.pytool&data=04%7C01%7Cbret.barkelew%40microsoft.com%7C28bf7b813a9a4d6e365508d93cf3fd8a%7C72f988bf86f141af91ab2d7cd011db47%7C1%7C0%7C637607840913338106%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C1000&sdata=v%2FqnYohY9bDCIAC8ZmqPkV3RvhaIuyQEXfkY9JNyIqs%3D&reserved=0>). I tested it in my local. It works well in Windows but does not work in Linux. It looks the NuGet in pytool only has windows version (C:\steven\unit_test_env\Lib\site-packages\edk2toolext\bin\NuGet.exe) but not support the Linux. Below is my detail test steps. Do the edk2 CI unit tests work in Linux?



  1.  Windows :

c:\steven>py -m venv unit_test_env

c:\steven>cd  c:\steven\edk2

c:\steven\edk2>c:\steven\unit_test_env\Scripts\activate.bat

(unit_test_env) c:\steven\edk2>pip install --upgrade -r pip-requirements.txt

(unit_test_env) c:\steven\edk2>stuart_setup -c .pytool/CISettings.py TOOL_CHAIN_TAG=VS2019

(unit_test_env) c:\steven\edk2>stuart_update -c .pytool/CISettings.py TOOL_CHAIN_TAG=VS2019

(unit_test_env) c:\steven\edk2>edksetup.bat

(unit_test_env) c:\steven\edk2>edksetup.bat Rebuild

(unit_test_env) c:\steven\edk2>stuart_ci_build -c .pytool/CISettings.py TOOL_CHAIN_TAG=VS2019 -a X64,IA32 -t DEBUG --verbose

...

PROGRESS - Overall Build Status: Success

SECTION - Summary

PROGRESS - Success



  1.  Linux:

jshi19 at ub2-uefi-b01:~/wksp_efi$ python3 -m venv unit_test_env

jshi19 at ub2-uefi-b01:~/wksp_efi$ source unit_test_env/bin/activate

(unit_test_env) jshi19 at ub2-uefi-b01:~/wksp_efi/edk2-2$ pip install --upgrade -r pip-requirements.txt

(unit_test_env) jshi19 at ub2-uefi-b01:~/wksp_efi/edk2-2$ make -C BaseTools/

(unit_test_env) jshi19 at ub2-uefi-b01:~/wksp_efi/edk2-2$ stuart_setup -c .pytool/CISettings.py TOOL_CHAIN_TAG=GCC5

(unit_test_env) jshi19 at ub2-uefi-b01:~/wksp_efi/edk2-2$ stuart_update -c .pytool/CISettings.py TOOL_CHAIN_TAG=GCC5

SECTION - Init SDE

WARNING - Using Pip Tools based BaseTools

SECTION - Loading Plugins

SECTION - Start Invocable Tool

SECTION - Initial update of environment

UpdatingWARNING - [SDE] Failed to fetch NugetDependecy: edk2-acpica-iasl at 20200717.0.0<mailto:edk2-acpica-iasl at 20200717.0.0>: [Nuget] We failed to install this version 20200717.0.0 of edk2-acpica-iasl

WARNING - [SDE] Failed to fetch NugetDependecy: mu_nasm at 2.15.05<mailto:mu_nasm at 2.15.05>: [Nuget] We failed to install this version 2.15.05 of mu_nasm

. Done

SECTION -       Updated/Verified 3 dependencies

SECTION - Second pass update of environment

UpdatingWARNING - [SDE] Failed to fetch NugetDependecy: edk2-acpica-iasl at 20200717.0.0<mailto:edk2-acpica-iasl at 20200717.0.0>: [Nuget] We failed to install this version 20200717.0.0 of edk2-acpica-iasl

WARNING - [SDE] Failed to fetch NugetDependecy: mu_nasm at 2.15.05<mailto:mu_nasm at 2.15.05>: [Nuget] We failed to install this version 2.15.05 of mu_nasm

. Done

SECTION -       Updated/Verified 3 dependencies

ERROR - We were unable to successfully update 2 dependencies in environment

SECTION - Summary

ERROR - Error









Thanks



Steven Shi










-=-=-=-=-=-=-=-=-=-=-=-
Groups.io Links: You receive all messages sent to this group.
View/Reply Online (#77431): https://edk2.groups.io/g/devel/message/77431
Mute This Topic: https://groups.io/mt/83873292/1813853
Group Owner: devel+owner at edk2.groups.io
Unsubscribe: https://edk2.groups.io/g/devel/unsub [edk2-devel-archive at redhat.com]
-=-=-=-=-=-=-=-=-=-=-=-


-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://listman.redhat.com/archives/edk2-devel-archive/attachments/20210702/20c675d5/attachment.htm>
-------------- next part --------------
A non-text attachment was scrubbed...
Name: image005.png
Type: image/png
Size: 195 bytes
Desc: image005.png
URL: <http://listman.redhat.com/archives/edk2-devel-archive/attachments/20210702/20c675d5/attachment.png>
-------------- next part --------------
A non-text attachment was scrubbed...
Name: image006.png
Type: image/png
Size: 195 bytes
Desc: image006.png
URL: <http://listman.redhat.com/archives/edk2-devel-archive/attachments/20210702/20c675d5/attachment-0001.png>

More information about the edk2-devel-archive mailing list