<html xmlns:v="urn:schemas-microsoft-com:vml" xmlns:o="urn:schemas-microsoft-com:office:office" xmlns:w="urn:schemas-microsoft-com:office:word" xmlns:m="http://schemas.microsoft.com/office/2004/12/omml" xmlns="http://www.w3.org/TR/REC-html40"> <head> <meta http-equiv="Content-Type" content="text/html; charset=utf-8"> <meta name="Generator" content="Microsoft Word 15 (filtered medium)"> <style></style> </head> <body bgcolor="white" lang="ZH-CN" link="blue" vlink="purple"> <div class="WordSection1"> For your question, I would suggest to use the second form, this is consistent with qemu, and will not bring platform specific knowledge to libvirt layer:<o:p></o:p> 2. or we let user specify the host-nodes. <memoryBacking> <mcdram nodeset="3-4", host-nodes="0-1"/> </memoryBacking> </domain><o:p></o:p> <a name="_MailEndCompose"><o:p> </o:p></a> Regards,<o:p></o:p> Dolpher<o:p></o:p> <div style="border:none;border-left:solid blue 1.5pt;padding:0cm 0cm 0cm 4.0pt"> <div> <div style="border:none;border-top:solid #E1E1E1 1.0pt;padding:3.0pt 0cm 0cm 0cm"> <a name="_____replyseparator"></a>From: Feng, Shaohe Sent: Monday, December 26, 2016 7:58 PM To: Daniel P. Berrange <berrange@redhat.com> Cc: Du, Dolpher <dolpher.du@intel.com>; He Chen <he.chen@linux.intel.com>; libvir-list@redhat.com; Zyskowski, Robert <robert.zyskowski@intel.com>; Daniluk, Lukasz <lukasz.daniluk@intel.com>; Zang, Rui <rui.zang@intel.com>; jdenemar@redhat.com Subject: Re: [libvirt] [RFC] phi support in libvirt<o:p></o:p> </div> </div> <o:p> </o:p> <div> Ｔｈａｎｋｓ, Daniel. So ｈｏｗ about: for the NUMA format, we still uses "memory" to describe the mcdram. But we remove the cpus elements. <numa> <cell id='3' memory='8' unit='GiB'/> </numa> <cell id='4' memory='8' unit='GiB'/> </numa> At present, for this kind CPUless NUMA , we only support mcdram as memroy backend. <domain> ... <memoryBacking> <mcdram nodeset="3-4"/> </memoryBacking> </domain> And we reject a CPUless NUMA without memroy backend. Maybe we will allow it in futures after qemu can handle it well. A question: 1. Should libvirt probe the "host-nodes" for this kind of memory to make a smart map? The qemu arguments will be as follow: -object memory-backend-ram,size=8G,prealloc=yes,host-nodes=0,policy=bind,id=node3 \ -numa node,nodeid=3,memdev=node3 \ -object memory-backend-ram,size=8G,prealloc=yes,host-nodes=0,policy=bind,id=node4 \ -numa node,nodeid=4,memdev=node4 \ 2. or we let user specify the host-nodes. <memoryBacking> <mcdram nodeset="3-4", host-nodes="0-1"/> </memoryBacking> </domain> BR ShaoHe Feng On 2016年12月21日 18:25, Daniel P. Berrange wrote:<o:p></o:p> </div> <blockquote style="margin-top:5.0pt;margin-bottom:5.0pt"> <pre>On Wed, Dec 21, 2016 at 12:51:29PM +0800, Feng, Shaohe wrote:<o:p></o:p></pre> <blockquote style="margin-top:5.0pt;margin-bottom:5.0pt"> <pre>Thanks. Dolpher.<o:p></o:p></pre> <pre><o:p> </o:p></pre> <pre>Reply inline.<o:p></o:p></pre> <pre><o:p> </o:p></pre> <pre><o:p> </o:p></pre> <pre>On 2016年12月21日 11:56, Du, Dolpher wrote:<o:p></o:p></pre> <blockquote style="margin-top:5.0pt;margin-bottom:5.0pt"> <pre>Shaohe was dropped from the loop, adding him back.<o:p></o:p></pre> <pre><o:p> </o:p></pre> <blockquote style="margin-top:5.0pt;margin-bottom:5.0pt"> <pre>-----Original Message-----<o:p></o:p></pre> <pre>From: He Chen [<a href="mailto:he.chen@linux.intel.com">mailto:he.chen@linux.intel.com</a>]<o:p></o:p></pre> <pre>Sent: Friday, December 9, 2016 3:46 PM<o:p></o:p></pre> <pre>To: Daniel P. Berrange <a href="mailto:berrange@redhat.com"><berrange@redhat.com></a><o:p></o:p></pre> <pre>Cc: <a href="mailto:libvir-list@redhat.com">libvir-list@redhat.com</a>; Du, Dolpher <a href="mailto:dolpher.du@intel.com"><dolpher.du@intel.com></a>; Zyskowski,<o:p></o:p></pre> <pre>Robert <a href="mailto:robert.zyskowski@intel.com"><robert.zyskowski@intel.com></a>; Daniluk, Lukasz<o:p></o:p></pre> <pre><a href="mailto:lukasz.daniluk@intel.com"><lukasz.daniluk@intel.com></a>; Zang, Rui <a href="mailto:rui.zang@intel.com"><rui.zang@intel.com></a>;<o:p></o:p></pre> <pre><a href="mailto:jdenemar@redhat.com">jdenemar@redhat.com</a><o:p></o:p></pre> <pre>Subject: Re: [libvirt] [RFC] phi support in libvirt<o:p></o:p></pre> <pre><o:p> </o:p></pre> <blockquote style="margin-top:5.0pt;margin-bottom:5.0pt"> <pre>On Mon, Dec 05, 2016 at 04:12:22PM +0000, Feng, Shaohe wrote:<o:p></o:p></pre> <blockquote style="margin-top:5.0pt;margin-bottom:5.0pt"> <pre>Hi all:<o:p></o:p></pre> <pre><o:p> </o:p></pre> <pre>As we are know Intel® Xeon phi targets high-performance computing and<o:p></o:p></pre> <pre>other parallel workloads.<o:p></o:p></pre> <pre>Now qemu has supported phi virtualization，it is time for libvirt to<o:p></o:p></pre> <pre>support phi.<o:p></o:p></pre> </blockquote> <pre>Can you provide pointer to the relevant QEMU changes.<o:p></o:p></pre> <pre><o:p> </o:p></pre> </blockquote> <pre>Xeon Phi Knights Landing (KNL) contains 2 primary hardware features, one<o:p></o:p></pre> <pre>is up to 288 CPUs which needs patches to support and we are pushing it,<o:p></o:p></pre> <pre>the other is Multi-Channel DRAM (MCDRAM) which does not need any changes<o:p></o:p></pre> <pre>currently.<o:p></o:p></pre> <pre><o:p> </o:p></pre> <pre>Let me introduce more about MCDRAM, MCDRAM is on-package<o:p></o:p></pre> <pre>high-bandwidth<o:p></o:p></pre> <pre>memory (~500GB/s).<o:p></o:p></pre> <pre><o:p> </o:p></pre> <pre>On KNL platform, hardware expose MCDRAM as a seperate, CPUless and<o:p></o:p></pre> <pre>remote NUMA node to OS so that MCDRAM will not be allocated by default<o:p></o:p></pre> <pre>(since MCDRAM node has no CPU, every CPU regards MCDRAM node as<o:p></o:p></pre> <pre>remote<o:p></o:p></pre> <pre>node). In this way, MCDRAM can be reserved for certain specific<o:p></o:p></pre> <pre>applications.<o:p></o:p></pre> <pre><o:p> </o:p></pre> <blockquote style="margin-top:5.0pt;margin-bottom:5.0pt"> <blockquote style="margin-top:5.0pt;margin-bottom:5.0pt"> <pre>Different from the traditional X86 server, There is a special numa<o:p></o:p></pre> <pre>node with Multi-Channel DRAM (MCDRAM) on Phi, but without any CPU .<o:p></o:p></pre> <pre><o:p> </o:p></pre> <pre>Now libvirt requires nonempty cpus argument for NUMA node, such as.<o:p></o:p></pre> <pre><numa><o:p></o:p></pre> <pre> <cell id='0' cpus='0-239' memory='80' unit='GiB'/><o:p></o:p></pre> <pre> <cell id='1' cpus='240-243' memory='16' unit='GiB'/> </numa><o:p></o:p></pre> <pre><o:p> </o:p></pre> <pre>In order to support phi virtualization, libvirt needs to allow a numa<o:p></o:p></pre> <pre>cell definition without 'cpu' attribution.<o:p></o:p></pre> <pre><o:p> </o:p></pre> <pre>Such as:<o:p></o:p></pre> <pre><numa><o:p></o:p></pre> <pre> <cell id='0' cpus='0-239' memory='80' unit='GiB'/><o:p></o:p></pre> <pre> <cell id='1' memory='16' unit='GiB'/> </numa><o:p></o:p></pre> <pre><o:p> </o:p></pre> <pre>When a cell without 'cpu', qemu will allocate memory by default MCDRAM<o:p></o:p></pre> </blockquote> </blockquote> <pre>instead of DDR.<o:p></o:p></pre> <blockquote style="margin-top:5.0pt;margin-bottom:5.0pt"> <pre>There's separate concepts at play which your description here is mixing up.<o:p></o:p></pre> <pre><o:p> </o:p></pre> <pre>First is the question of whether the guest NUMA node can be created with<o:p></o:p></pre> </blockquote> <pre>only RAM or CPUs, or a mix of both.<o:p></o:p></pre> <blockquote style="margin-top:5.0pt;margin-bottom:5.0pt"> <pre>Second is the question of what kind of host RAM (MCDRAM vs DDR) is used<o:p></o:p></pre> </blockquote> <pre>as the backing store for the guest<o:p></o:p></pre> <pre>Guest NUMA node shoulde be created with memory only (keep the same as<o:p></o:p></pre> <pre>host's) and the more important things is the memory should bind to (come<o:p></o:p></pre> <pre>from) host MCDRAM node.<o:p></o:p></pre> </blockquote> </blockquote> <pre>So I suggest libvirt distinguish the MCDRAM<o:p></o:p></pre> <pre><o:p> </o:p></pre> <pre>And the MCDRAM numa config as follow, add a "mcdram" attribute for "cell"<o:p></o:p></pre> <pre>element:<o:p></o:p></pre> <pre><numa><o:p></o:p></pre> <pre> <cell id='1' mcdram='16' unit='GiB'/> </numa><o:p></o:p></pre> <pre> <cell id='0' cpus='0-239' memory='80' unit='GiB'/><o:p></o:p></pre> </blockquote> <pre><o:p> </o:p></pre> <pre>No, that is not backwards compatible for applications using libvirt.<o:p></o:p></pre> <pre><o:p> </o:p></pre> <pre>We already have a place for storing info about memory backing type,<o:p></o:p></pre> <pre>which we use for huge pages. mcdram should use the same approach<o:p></o:p></pre> <pre>IMHO. eg<o:p></o:p></pre> <pre><o:p> </o:p></pre> <pre><domain><o:p></o:p></pre> <pre> ...<o:p></o:p></pre> <pre> <memoryBacking><o:p></o:p></pre> <pre> <mcdram nodeset="3-4"/><o:p></o:p></pre> <pre> </memoryBacking><o:p></o:p></pre> <pre></domain><o:p></o:p></pre> <pre><o:p> </o:p></pre> <pre>to indicate that nodes 3 & 4 should use mcdram<o:p></o:p></pre> <pre><o:p> </o:p></pre> <pre>Regards,<o:p></o:p></pre> <pre>Daniel<o:p></o:p></pre> </blockquote> <o:p> </o:p> </div> </div> </body> </html>