<html> <head> <meta content="text/html; charset=utf-8" http-equiv="Content-Type"> </head> <body bgcolor="#FFFFFF" text="#000000"> <div class="moz-cite-prefix">Ｔｈａｎｋｓ, Daniel. So ｈｏｗ about: for the <meta http-equiv="content-type" content="text/html; charset=utf-8"> <span style="color: rgb(0, 0, 0); font-family: LibvirtOverpass; font-size: medium; font-style: normal; font-variant-ligatures: normal; font-variant-caps: normal; font-weight: normal; letter-spacing: normal; orphans: 2; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; widows: 2; word-spacing: 0px; -webkit-text-stroke-width: 0px; background-color: rgb(255, 255, 255); display: inline !important; float: none;">NUMA format, we still uses "memory" to describe the mcdram. But we remove the cpus elements. <numa> <cell id='3' memory='8' unit='GiB'/> </numa> <cell id='4' memory='8' unit='GiB'/> </numa> At present, for this kind CPUless <span style="color: rgb(0, 0, 0); font-family: LibvirtOverpass; font-size: medium; font-style: normal; font-variant-ligatures: normal; font-variant-caps: normal; font-weight: normal; letter-spacing: normal; orphans: 2; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; widows: 2; word-spacing: 0px; -webkit-text-stroke-width: 0px; background-color: rgb(255, 255, 255); display: inline !important; float: none;">NUMA , we only support mcdram as memroy backend. <domain> ... <memoryBacking> <mcdram nodeset="3-4"/> </memoryBacking> </domain> And we reject a CPUless <span style="color: rgb(0, 0, 0); font-family: LibvirtOverpass; font-size: medium; font-style: normal; font-variant-ligatures: normal; font-variant-caps: normal; font-weight: normal; letter-spacing: normal; orphans: 2; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; widows: 2; word-spacing: 0px; -webkit-text-stroke-width: 0px; background-color: rgb(255, 255, 255); display: inline !important; float: none;">NUMA without memroy backend. Maybe we will allow it in futures after qemu can handle it well. A question: 1. Should libvirt probe the "host-nodes" for this kind of memory to make a smart map? The qemu arguments will be as follow: -object memory-backend-ram,size=8G,prealloc=yes,host-nodes=0,policy=bind,id=node3 \ -numa node,nodeid=3,memdev=node3 \ -object memory-backend-ram,size=8G,prealloc=yes,host-nodes=0,policy=bind,id=node4 \ -numa node,nodeid=4,memdev=node4 \ 2. or we let user specify the host-nodes. <memoryBacking> <mcdram nodeset="3-4", host-nodes="0-1"/> </memoryBacking> </domain> BR ShaoHe Feng On 2016年12月21日 18:25, Daniel P. Berrange wrote: </div> <blockquote cite="mid:20161221102546.GA9460@redhat.com" type="cite"> <pre wrap="">On Wed, Dec 21, 2016 at 12:51:29PM +0800, Feng, Shaohe wrote: </pre> <blockquote type="cite"> <pre wrap="">Thanks. Dolpher. Reply inline. On 2016年12月21日 11:56, Du, Dolpher wrote: </pre> <blockquote type="cite"> <pre wrap="">Shaohe was dropped from the loop, adding him back. </pre> <blockquote type="cite"> <pre wrap="">-----Original Message----- From: He Chen [<a class="moz-txt-link-freetext" href="mailto:he.chen@linux.intel.com">mailto:he.chen@linux.intel.com</a>] Sent: Friday, December 9, 2016 3:46 PM To: Daniel P. Berrange <a class="moz-txt-link-rfc2396E" href="mailto:berrange@redhat.com"><berrange@redhat.com></a> Cc: <a class="moz-txt-link-abbreviated" href="mailto:libvir-list@redhat.com">libvir-list@redhat.com</a>; Du, Dolpher <a class="moz-txt-link-rfc2396E" href="mailto:dolpher.du@intel.com"><dolpher.du@intel.com></a>; Zyskowski, Robert <a class="moz-txt-link-rfc2396E" href="mailto:robert.zyskowski@intel.com"><robert.zyskowski@intel.com></a>; Daniluk, Lukasz <a class="moz-txt-link-rfc2396E" href="mailto:lukasz.daniluk@intel.com"><lukasz.daniluk@intel.com></a>; Zang, Rui <a class="moz-txt-link-rfc2396E" href="mailto:rui.zang@intel.com"><rui.zang@intel.com></a>; <a class="moz-txt-link-abbreviated" href="mailto:jdenemar@redhat.com">jdenemar@redhat.com</a> Subject: Re: [libvirt] [RFC] phi support in libvirt </pre> <blockquote type="cite"> <pre wrap="">On Mon, Dec 05, 2016 at 04:12:22PM +0000, Feng, Shaohe wrote: </pre> <blockquote type="cite"> <pre wrap="">Hi all: As we are know Intel® Xeon phi targets high-performance computing and other parallel workloads. Now qemu has supported phi virtualization，it is time for libvirt to support phi. </pre> </blockquote> <pre wrap="">Can you provide pointer to the relevant QEMU changes. </pre> </blockquote> <pre wrap="">Xeon Phi Knights Landing (KNL) contains 2 primary hardware features, one is up to 288 CPUs which needs patches to support and we are pushing it, the other is Multi-Channel DRAM (MCDRAM) which does not need any changes currently. Let me introduce more about MCDRAM, MCDRAM is on-package high-bandwidth memory (~500GB/s). On KNL platform, hardware expose MCDRAM as a seperate, CPUless and remote NUMA node to OS so that MCDRAM will not be allocated by default (since MCDRAM node has no CPU, every CPU regards MCDRAM node as remote node). In this way, MCDRAM can be reserved for certain specific applications. </pre> <blockquote type="cite"> <blockquote type="cite"> <pre wrap="">Different from the traditional X86 server, There is a special numa node with Multi-Channel DRAM (MCDRAM) on Phi, but without any CPU . Now libvirt requires nonempty cpus argument for NUMA node, such as. <numa> <cell id='0' cpus='0-239' memory='80' unit='GiB'/> <cell id='1' cpus='240-243' memory='16' unit='GiB'/> </numa> In order to support phi virtualization, libvirt needs to allow a numa cell definition without 'cpu' attribution. Such as: <numa> <cell id='0' cpus='0-239' memory='80' unit='GiB'/> <cell id='1' memory='16' unit='GiB'/> </numa> When a cell without 'cpu', qemu will allocate memory by default MCDRAM </pre> </blockquote> </blockquote> <pre wrap="">instead of DDR. </pre> <blockquote type="cite"> <pre wrap="">There's separate concepts at play which your description here is mixing up. First is the question of whether the guest NUMA node can be created with </pre> </blockquote> <pre wrap="">only RAM or CPUs, or a mix of both. </pre> <blockquote type="cite"> <pre wrap="">Second is the question of what kind of host RAM (MCDRAM vs DDR) is used </pre> </blockquote> <pre wrap="">as the backing store for the guest Guest NUMA node shoulde be created with memory only (keep the same as host's) and the more important things is the memory should bind to (come from) host MCDRAM node. </pre> </blockquote> </blockquote> <pre wrap="">So I suggest libvirt distinguish the MCDRAM And the MCDRAM numa config as follow, add a "mcdram" attribute for "cell" element: <numa> <cell id='1' mcdram='16' unit='GiB'/> </numa> <cell id='0' cpus='0-239' memory='80' unit='GiB'/> </pre> </blockquote> <pre wrap=""> No, that is not backwards compatible for applications using libvirt. We already have a place for storing info about memory backing type, which we use for huge pages. mcdram should use the same approach IMHO. eg <domain> ... <memoryBacking> <mcdram nodeset="3-4"/> </memoryBacking> </domain> to indicate that nodes 3 & 4 should use mcdram Regards, Daniel </pre> </blockquote> </body> </html>