<html> <head> <meta http-equiv="Content-Type" content="text/html; charset=iso-8859-1"> <style type="text/css" style="display:none;"> P {margin-top:0;margin-bottom:0;} </style> </head> <body dir="ltr"> <div style="font-family: Calibri, Arial, Helvetica, sans-serif; font-size: 12pt; color: rgb(0, 0, 0);"> Hello Ard and Marc,</div> <div style="font-family: Calibri, Arial, Helvetica, sans-serif; font-size: 12pt; color: rgb(0, 0, 0);"> <br> </div> <div style="font-family: Calibri, Arial, Helvetica, sans-serif; font-size: 12pt; color: rgb(0, 0, 0);"> I apologize for not providing the background on this in the commit message and I understand the commit message is not very clear as well. Let me try to summarize the problem.</div> <div style="font-family: Calibri, Arial, Helvetica, sans-serif; font-size: 12pt; color: rgb(0, 0, 0);"> <br> </div> <div style="font-family: Calibri, Arial, Helvetica, sans-serif; font-size: 12pt; color: rgb(0, 0, 0);"> <span style="margin:0px;font-size:12pt;font-family:Calibri, Arial, Helvetica, sans-serif, serif, EmojiFont;color:black;background-color:rgb(255, 255, 255)">In our UEFI implementation, we are doing the following as part of the initial MMU setup:</span></div> <div style="font-family: Calibri, Arial, Helvetica, sans-serif; font-size: 12pt; color: rgb(0, 0, 0);"> <ol> <li style="font-size:12pt;font-family:Calibri, Arial, Helvetica, sans-serif, serif, EmojiFont;color:black"> <span><span style="margin:0px;font-size:12pt;font-family:Calibri, Arial, Helvetica, sans-serif, serif, EmojiFont;color:black;background-color:rgb(255, 255, 255)">Set the applicable device memory as nGnRnE memory.</span></span></li><li style="font-size:12pt;font-family:Calibri, Arial, Helvetica, sans-serif, serif, EmojiFont;color:black"> <span><span style="margin:0px;font-size:12pt;font-family:Calibri, Arial, Helvetica, sans-serif, serif, EmojiFont;color:black;background-color:rgb(255, 255, 255)">Set the whole DRAM as normal memory that translates to RW and executable memory.</span></span></li><li style="font-size:12pt;font-family:Calibri, Arial, Helvetica, sans-serif, serif, EmojiFont;color:black"> <span><span style="margin:0px;font-size:12pt;font-family:Calibri, Arial, Helvetica, sans-serif, serif, EmojiFont;color:black;background-color:rgb(255, 255, 255)">Enable caches and MMU.</span></span></li><li style="font-size:12pt;font-family:Calibri, Arial, Helvetica, sans-serif, serif, EmojiFont;color:black"> <span><span style="margin:0px;font-size:12pt;font-family:Calibri, Arial, Helvetica, sans-serif, serif, EmojiFont;color:black;background-color:rgb(255, 255, 255)">At this time, the memory map looks correct when I check that from DS-5.</span></span></li></ol> <div><span style="color: black; font-family: Calibri, Arial, Helvetica, sans-serif, serif, EmojiFont; font-size: 12pt;">When we start dispatching drivers, DxeCore dispatches a driver and marks its code area as RO and executable and its data region as RW and non-executable. What we are seeing randomly is that some of the page tables (using DS-5) have invalid output address that leads to the correct input address from UEFI being translated to an unavailable memory location causing a crash sometime in EL2 or sometimes as a RAS error in EL3.</span></div> <div><span style="color: black; font-family: Calibri, Arial, Helvetica, sans-serif, serif, EmojiFont; font-size: 12pt;"><br> </span></div> </div> <div style="font-family: Calibri, Arial, Helvetica, sans-serif; font-size: 12pt; color: rgb(0, 0, 0);"> <div style="margin:0px;font-size:12pt;font-family:Calibri, Arial, Helvetica, sans-serif, serif, EmojiFont;color:black;background-color:rgb(255, 255, 255)"> When I reached out to the CPU team here, they said <span style="font-variant-ligatures:common-ligatures;text-align:left;background-color:rgb(255, 255, 255);display:inline !important">Arm® Cortex®-A78AE is a highly speculative core and we need to have appropriate barriers in place so that there is consistency in the way an address is accessed especially if it is done right after there is a change in translation tables. Based on this, I started some experimentation wrt caches whenever MMUs are enabled and I found that invalidating the instruction cache after enabling MMUs solves this problem.</span></div> <div style="margin:0px;font-size:12pt;font-family:Calibri, Arial, Helvetica, sans-serif, serif, EmojiFont;color:black;background-color:rgb(255, 255, 255)"> <br> </div> <div style="margin:0px;font-size:12pt;font-family:Calibri, Arial, Helvetica, sans-serif, serif, EmojiFont;color:black;background-color:rgb(255, 255, 255)"> Please note that I could be wrong with my hypothesis here and I may just be masking the issue. If that is the case, please let me know what I should be trying as I am out of ideas at this point. Also, <span style="color: black; font-family: Calibri, Arial, Helvetica, sans-serif, serif, EmojiFont; font-size: 12pt;">the same UEFI works on NVIDIA's Xavier Silicon that has Carmel cores but shows this issue on Orin Silicon that has </span><span style="color: black; font-family: Calibri, Arial, Helvetica, sans-serif, serif, EmojiFont; font-size: 12pt; font-variant-ligatures: common-ligatures;">Arm® Cortex®-A78AE v8.2 64-bit CPU.</span></div> <div style="margin:0px;font-size:12pt;font-family:Calibri, Arial, Helvetica, sans-serif, serif, EmojiFont;color:black;background-color:rgb(255, 255, 255)"> <span style="color: black; font-family: Calibri, Arial, Helvetica, sans-serif, serif, EmojiFont; font-size: 12pt; font-variant-ligatures: common-ligatures;"><br> </span></div> <div style="margin:0px;font-size:12pt;font-family:Calibri, Arial, Helvetica, sans-serif, serif, EmojiFont;color:black;background-color:rgb(255, 255, 255)"> <span style="color: black; font-family: Calibri, Arial, Helvetica, sans-serif, serif, EmojiFont; font-size: 12pt; font-variant-ligatures: common-ligatures;"><a id="OWAAM165597" class="J9Y1oNF3ZpoR5LC3M2PHm mention ms-bgc-nlr ms-fcl-b" href="mailto:maz@kernel.org">@Marc Zyngier</a>,</span></div> <div style="margin:0px;font-size:12pt;font-family:Calibri, Arial, Helvetica, sans-serif, serif, EmojiFont;color:black;background-color:rgb(255, 255, 255)"> <span style="color: black; font-family: Calibri, Arial, Helvetica, sans-serif, serif, EmojiFont; font-size: 12pt; font-variant-ligatures: common-ligatures;"><br> </span></div> <div style="margin:0px;font-size:12pt;font-family:Calibri, Arial, Helvetica, sans-serif, serif, EmojiFont;color:black;background-color:rgb(255, 255, 255)"> <span style="color: black; font-family: Calibri, Arial, Helvetica, sans-serif, serif, EmojiFont; font-size: 12pt; font-variant-ligatures: common-ligatures;">I agree with your concern about using "dsb sy" and we should replace that with "dsb nsh" as we are running this only on a single core during boot.</span></div> <div style="margin:0px;font-size:12pt;font-family:Calibri, Arial, Helvetica, sans-serif, serif, EmojiFont;color:black;background-color:rgb(255, 255, 255)"> <span style="color: black; font-family: Calibri, Arial, Helvetica, sans-serif, serif, EmojiFont; font-size: 12pt; font-variant-ligatures: common-ligatures;"><br> </span></div> <div style="margin:0px;font-size:12pt;font-family:Calibri, Arial, Helvetica, sans-serif, serif, EmojiFont;color:black;background-color:rgb(255, 255, 255)"> <span style="color: black; font-family: Calibri, Arial, Helvetica, sans-serif, serif, EmojiFont; font-size: 12pt; font-variant-ligatures: common-ligatures;">Thanks</span></div> <div style="margin:0px;font-size:12pt;font-family:Calibri, Arial, Helvetica, sans-serif, serif, EmojiFont;color:black;background-color:rgb(255, 255, 255)"> <span style="color: black; font-family: Calibri, Arial, Helvetica, sans-serif, serif, EmojiFont; font-size: 12pt; font-variant-ligatures: common-ligatures;">Ashish</span></div> </div> <div id="appendonsend"></div> <hr style="display:inline-block;width:98%" tabindex="-1"> <div id="divRplyFwdMsg" dir="ltr"><font face="Calibri, sans-serif" style="font-size:11pt" color="#000000"><b>From:</b> Marc Zyngier <maz@kernel.org><br> <b>Sent:</b> Wednesday, February 23, 2022 1:58 AM<br> <b>To:</b> Ard Biesheuvel <ardb@kernel.org>; Ashish Singhal <ashishsingha@nvidia.com><br> <b>Cc:</b> edk2-devel-groups-io <devel@edk2.groups.io>; Sami Mujawar <sami.mujawar@arm.com>; Ard Biesheuvel <ardb+tianocore@kernel.org>; Leif Lindholm <quic_llindhol@quicinc.com><br> <b>Subject:</b> Re: [PATCH] ArmPkg: Invalidate Instruction Cache On MMU Enable</font> <div> </div> </div> <div class="BodyFragment"><font size="2"><span style="font-size:11pt;"> <div class="PlainText">External email: Use caution opening links or attachments<br> <br> <br> On Wed, 23 Feb 2022 07:02:28 +0000,<br> Ard Biesheuvel <ardb@kernel.org> wrote:<br> ><br> > (+ Marc)<br> ><br> > On Tue, 22 Feb 2022 at 03:42, Ashish Singhal <ashishsingha@nvidia.com> wrote:<br> > ><br> > > Even with MMU turned off, instruction cache can speculate<br> > > and fetch instructions. This can cause a crash if region<br> > > being executed has been modified recently. With this patch,<br> > > we ensure that instruction cache is invalidated right after<br> > > MMU has been enabled and any potentially stale instruction<br> > > fetched earlier has been discarded.<br> <br> Are you doing code patching while the MMU is off?<br> <br> > ><br> ><br> > I don't follow this reasoning. Every time the UEFI image loader loads<br> > an image into memory, it does so with MMU and caches enabled, and the<br> > code regions are cleaned to the PoU before being invalidated in the<br> > I-cache. So how could these regions be stale? The only code that needs<br> > special care here is the little trampoline that executes with the MMU<br> > off, but this is being taken care of explicitly, by cleaning the code<br> > to the PoC before invoking it.<br> ><br> > > This is specially helpful when the memory attributes of a<br> > > region in MMU are being changed and some instructions<br> > > operating on the region are prefetched in the instruction<br> > > cache.<br> > ><br> <br> Huh, this is way too vague. How do you expect anyone to understand<br> your problem based on this?<br> <br> ><br> > This sounds like a TLB problem not a cache problem. For the sake of<br> > posterity, could you include a more detailed description of the issue<br> > in the commit log? IIRC, this is about speculative instruction fetches<br> > hitting device memory locations?<br> ><br> > As I mentioned before, the architecture does not permit speculative<br> > instruction fetches for regions mapped with the XN attribute.<br> ><br> > Is the issue occurring under virtualization? Is there a stage 2<br> > mapping that lacks the XN attribute for the device memory region in<br> > question?<br> ><br> > > Signed-off-by: Ashish Singhal <ashishsingha@nvidia.com><br> > > ---<br> > > ArmPkg/Library/ArmLib/AArch64/AArch64Support.S | 4 +++-<br> > > ArmPkg/Library/ArmMmuLib/AArch64/ArmMmuLibReplaceEntry.S | 2 ++<br> > > 2 files changed, 5 insertions(+), 1 deletion(-)<br> > ><br> > > diff --git a/ArmPkg/Library/ArmLib/AArch64/AArch64Support.S b/ArmPkg/Library/ArmLib/AArch64/AArch64Support.S<br> > > index d3cc1e8671..9648245182 100644<br> > > --- a/ArmPkg/Library/ArmLib/AArch64/AArch64Support.S<br> > > +++ b/ArmPkg/Library/ArmLib/AArch64/AArch64Support.S<br> > > @@ -89,7 +89,9 @@ ASM_FUNC(ArmEnableMmu)<br> > > dsb nsh<br> > > isb<br> > > msr sctlr_el3, x0 // Write back<br> > > -4: isb<br> > > +4: ic iallu<br> > > + dsb sy<br> <br> Why DSB SY? Given that you are only invalidating on a single CPU,<br> DSB NSH should be enough.<br> <br> > > + isb<br> > > ret<br> > ><br> > ><br> > > diff --git a/ArmPkg/Library/ArmMmuLib/AArch64/ArmMmuLibReplaceEntry.S b/ArmPkg/Library/ArmMmuLib/AArch64/ArmMmuLibReplaceEntry.S<br> > > index 66ebca571e..56cc2dd73f 100644<br> > > --- a/ArmPkg/Library/ArmMmuLib/AArch64/ArmMmuLibReplaceEntry.S<br> > > +++ b/ArmPkg/Library/ArmMmuLib/AArch64/ArmMmuLibReplaceEntry.S<br> > > @@ -37,6 +37,8 @@<br> > ><br> > > // re-enable the MMU<br> > > msr sctlr_el\el, x8<br> > > + ic iallu<br> > > + dsb sy<br> <br> Same thing here.<br> <br> M.<br> <br> --<br> Without deviation from the norm, progress is not possible.<br> </div> </span></font></div> </body> </html> <div width="1" style="color:white;clear:both">_._,_._,_</div> <hr> Groups.io Links:<p> You receive all messages sent to this group. <p> <a target="_blank" href="https://edk2.groups.io/g/devel/message/86917">View/Reply Online (#86917)</a> | | <a target="_blank" href="https://groups.io/mt/89309504/1813853">Mute This Topic</a> | <a href="https://edk2.groups.io/g/devel/post">New Topic</a><br> <a href="https://edk2.groups.io/g/devel/editsub/1813853">Your Subscription</a> | <a href="mailto:devel+owner@edk2.groups.io">Contact Group Owner</a> | <a href="https://edk2.groups.io/g/devel/unsub">Unsubscribe</a> [edk2-devel-archive@redhat.com]<br> <div width="1" style="color:white;clear:both">_._,_._,_</div>