[EXTERNAL] Re: [edk2-devel] generating the test matrix for local CI runs

Thu Mar 4 18:45:32 UTC 2021

On 03/04/21 19:20, Bret Barkelew wrote:
> Thinking further, it will skip some tests if you haven’t installed the prerequisites (e.g. spell check). We could potentially add a single script to install the prerequisites, but then there are the prereqs for the prereqs (e.g. Node).
> 
> To more clearly address your question, the “tests” that a GitHub CI is broken up into are arbitrary subdivisions of the tests that run locally (when you run “stuart_ci_build” without parameters) so that the workload can be divided amongst the available worker agents. If you were to compare the individual test logs against your local, monolithic test log, you would find that all cases were covered in both (assuming the aforementioned prereqs are installed).

Thank you -- I hope to test this later.

(1) A question about deps: if a dep is missing, can I force an error
(rather than a silent skip)?

Again, I distinctly remember doing what I then considered "due
diligence" locally, only to see github reject the PR later. If that was
because some deps were missing locally, I'd really like to turn that
scenario into a local error.

A script for installing all dependencies would help of course, but not
as a *replacement* for this kind of deps enforcement -- instead, only
for solving the missing dependency problems. I'd still want the local CI
to break hard if it couldn't offer the exact coverage as github.com.

Another question below:

> 
> - Bret
> 
> From: Bret Barkelew via groups.io<mailto:bret.barkelew=microsoft.com at groups.io>
> Sent: Thursday, March 4, 2021 9:57 AM
> To: devel at edk2.groups.io<mailto:devel at edk2.groups.io>; rebecca at nuviainc.com<mailto:rebecca at nuviainc.com>; lersek at redhat.com<mailto:lersek at redhat.com>
> Cc: Sean Brogan<mailto:sean.brogan at microsoft.com>; Kinney, Michael D<mailto:michael.d.kinney at intel.com>
> Subject: Re: [EXTERNAL] Re: [edk2-devel] generating the test matrix for local CI runs
> 
> *An* answer is:
> - There’s a DevOps virtual environment that you can run that will pretend to be a build agent. I played with it years ago, but that was before the CI tooling (and it was years ago) so I don’t know how the capabilities compare.
> 
> My real answer is:
> - If you run the three commands -- setup, update, and build -- without any parameters other than your TOOL_CHAIN_TAG, it should run all tests on all packages for that particular OS/toolchain.

(2) You mention "for that particular OS/toolchain".

How can I make sure that I'm using the exact same toolchain as github would?

I'd like to tell the tooling only the OS (Windows vs. Linux), and then
the toolchain should be auto-selected, same way as it is auto-selected
on github. For example, if I'm in my Windows 10 VM, then I should be
*unable* to request anything else than the latest VS toolchain;
similarly, in the Linux VM, I should be *unable* to ask for (e.g.) the
GCC49 toolchain; only the GCC50 toolchain should be usable.

When I submit a github PR, I'm not asked to pick toolchains for CI; can
we automate the toolchain selection locally too?

(Sorry if I misunderstood your comment above -- if you are saying that
it's permitted, but *not required*, to specify TOOL_CHAIN_TAG, then my
question (2) falls away.)

Thanks!
Laszlo
> 
> - Bret
> 
> From: Rebecca Cran via groups.io<mailto:rebecca=nuviainc.com at groups.io>
> Sent: Thursday, March 4, 2021 9:45 AM
> To: devel at edk2.groups.io<mailto:devel at edk2.groups.io>; lersek at redhat.com<mailto:lersek at redhat.com>
> Cc: Sean Brogan<mailto:sean.brogan at microsoft.com>; Bret Barkelew<mailto:Bret.Barkelew at microsoft.com>; Kinney, Michael D<mailto:michael.d.kinney at intel.com>
> Subject: [EXTERNAL] Re: [edk2-devel] generating the test matrix for local CI runs
> 
> On 3/4/21 10:04 AM, Laszlo Ersek wrote:
>> (Sigh, my address book pulled one over me -- resending to the correct
>> list address now. Please ignore the previous posting that you may have
>> gotten "in private".)
>>
>> Hi All,
>>
>> while I can run specific CI tests locally, using the commands listed at:
>>
>> https://nam06.safelinks.protection.outlook.com/?url=https%3A%2F%2Fgithub.com%2Ftianocore%2Fedk2%2Ftree%2Fmaster%2F.pytool&data=04%7C01%7Cbret.barkelew%40microsoft.com%7Ce4e5493578cf45b804c608d8df354b05%7C72f988bf86f141af91ab2d7cd011db47%7C1%7C0%7C637504767278995847%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C1000&sdata=l8WblhD%2B7dpkFaCZ8roSfNUmj1imytHi5eUT%2F%2BKrLCo%3D&reserved=0<https://nam06.safelinks.protection.outlook.com/?url=https%3A%2F%2Fgithub.com%2Ftianocore%2Fedk2%2Ftree%2Fmaster%2F.pytool&data=04%7C01%7CBret.Barkelew%40microsoft.com%7C385eef1d988f4a56155a08d8df36ecce%7C72f988bf86f141af91ab2d7cd011db47%7C1%7C0%7C637504774296518259%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C1000&sdata=UPVktVmDdChYhoc6zErhuz57Dllw21gqcwi9crhUdp8%3D&reserved=0>
>> https://nam06.safelinks.protection.outlook.com/?url=https%3A%2F%2Fgithub.com%2Ftianocore%2Fedk2%2Ftree%2Fmaster%2FArmVirtPkg%2FPlatformCI&data=04%7C01%7Cbret.barkelew%40microsoft.com%7Ce4e5493578cf45b804c608d8df354b05%7C72f988bf86f141af91ab2d7cd011db47%7C1%7C0%7C637504767278995847%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C1000&sdata=8yG7Mro9Li4XeCvfmPhfReBbipz47nyAjUmMKMGfh%2Bo%3D&reserved=0<https://nam06.safelinks.protection.outlook.com/?url=https%3A%2F%2Fgithub.com%2Ftianocore%2Fedk2%2Ftree%2Fmaster%2FArmVirtPkg%2FPlatformCI&data=04%7C01%7CBret.Barkelew%40microsoft.com%7C385eef1d988f4a56155a08d8df36ecce%7C72f988bf86f141af91ab2d7cd011db47%7C1%7C0%7C637504774296518259%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C1000&sdata=mcoSK9x7FSSJzlI86mdvs%2Btp7zdeF42IJn62mJxxozo%3D&reserved=0>
>> https://nam06.safelinks.protection.outlook.com/?url=https%3A%2F%2Fgithub.com%2Ftianocore%2Fedk2%2Ftree%2Fmaster%2FOvmfPkg%2FPlatformCI&data=04%7C01%7Cbret.barkelew%40microsoft.com%7Ce4e5493578cf45b804c608d8df354b05%7C72f988bf86f141af91ab2d7cd011db47%7C1%7C0%7C637504767278995847%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C1000&sdata=NUhAJ8zkITwPkvCrUqb7ZogSQTJuBXv0eL6Au0sYC2M%3D&reserved=0<https://nam06.safelinks.protection.outlook.com/?url=https%3A%2F%2Fgithub.com%2Ftianocore%2Fedk2%2Ftree%2Fmaster%2FOvmfPkg%2FPlatformCI&data=04%7C01%7CBret.Barkelew%40microsoft.com%7C385eef1d988f4a56155a08d8df36ecce%7C72f988bf86f141af91ab2d7cd011db47%7C1%7C0%7C637504774296528219%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C1000&sdata=zOv0z287ugh6PmcVk9EMAvxx4RE8U8iHUYemWzO1FaY%3D&reserved=0>
>>
>> I don't know how I can determine and drive the whole "test plan" locally.
>>
>> When I submit a PR to github, "something" generates (say) 83 test cases.
>> How can I run that "something" locally?
>>
>> Assume I have the following:
>> - a Linux builder VM, with up-to-date tools,
>> - a Windows builder VM, with up-to-date tools,
>> - a topic branch, on top of "master".
>>
>> How do I generate and run those same 83 tests, locally?
>>
>> It has happened to me that all my local CI steps succeeded (meaning both
>> the Windows and the Linux builder), but github still rejected the PR --
>> the reason was that I failed to think of a particular test that github
>> would run. Composing 83 "stuart" command lines isn't something I'd like
>> to do manually.
> 
> FWIW it looks like people have been looking for similar functionality of
> testing a pipeline locally, and not found it.
> 
> e.g.:
> https://nam06.safelinks.protection.outlook.com/?url=https%3A%2F%2Fgithub.com%2Fmicrosoft%2Fazure-pipelines-agent%2Fissues%2F1764&data=04%7C01%7Cbret.barkelew%40microsoft.com%7Ce4e5493578cf45b804c608d8df354b05%7C72f988bf86f141af91ab2d7cd011db47%7C1%7C0%7C637504767278995847%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C1000&sdata=XMxZesyKxVN9Wg3SyRpUbnnmKzojevHCqQhzQ1Pc4%2Bg%3D&reserved=0<https://nam06.safelinks.protection.outlook.com/?url=https%3A%2F%2Fgithub.com%2Fmicrosoft%2Fazure-pipelines-agent%2Fissues%2F1764&data=04%7C01%7CBret.Barkelew%40microsoft.com%7C385eef1d988f4a56155a08d8df36ecce%7C72f988bf86f141af91ab2d7cd011db47%7C1%7C0%7C637504774296528219%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C1000&sdata=uwpRNBUe4wFq7M0tRXBs8U4O1Qn6T0jf76Acr7n0qu8%3D&reserved=0>
> https://nam06.safelinks.protection.outlook.com/?url=https%3A%2F%2Fdevelopercommunity.visualstudio.com%2Ft%2Fyaml-how-to-test-yaml-locally-before-commit%2F1302759&data=04%7C01%7Cbret.barkelew%40microsoft.com%7Ce4e5493578cf45b804c608d8df354b05%7C72f988bf86f141af91ab2d7cd011db47%7C1%7C0%7C637504767278995847%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C1000&sdata=gwGZAQJAIXIS0LODFpglrCf5Ua2JeMnNm3A3auCrhx8%3D&reserved=0<https://nam06.safelinks.protection.outlook.com/?url=https%3A%2F%2Fdevelopercommunity.visualstudio.com%2Ft%2Fyaml-how-to-test-yaml-locally-before-commit%2F1302759&data=04%7C01%7CBret.Barkelew%40microsoft.com%7C385eef1d988f4a56155a08d8df36ecce%7C72f988bf86f141af91ab2d7cd011db47%7C1%7C0%7C637504774296538176%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C1000&sdata=ISHz%2FhaG88Cg6V%2BMwguTJNLrFwABVeNmOn4mq5meKd8%3D&reserved=0>
> 
> --
> Rebecca Cran
> 
> 
> 
> 
> 
> 
> 

-=-=-=-=-=-=-=-=-=-=-=-
Groups.io Links: You receive all messages sent to this group.
View/Reply Online (#72441): https://edk2.groups.io/g/devel/message/72441
Mute This Topic: https://groups.io/mt/81083112/1813853
Group Owner: devel+owner at edk2.groups.io
Unsubscribe: https://edk2.groups.io/g/devel/unsub [edk2-devel-archive at redhat.com]
-=-=-=-=-=-=-=-=-=-=-=-