<html>
<head>
<meta http-equiv="Content-Type" content="text/html; charset=us-ascii">
</head>
<body style="word-wrap: break-word; -webkit-nbsp-mode: space; -webkit-line-break: after-white-space; color: rgb(0, 0, 0); font-size: 14px; font-family: Calibri, sans-serif;">
<div>Thanks Alex! Will try to reduce the memory used and check.</div>
<div><br>
</div>
<div>Anjali</div>
<div><br>
</div>
<span id="OLK_SRC_BODY_SECTION">
<div style="font-family:Calibri; font-size:11pt; text-align:left; color:black; BORDER-BOTTOM: medium none; BORDER-LEFT: medium none; PADDING-BOTTOM: 0in; PADDING-LEFT: 0in; PADDING-RIGHT: 0in; BORDER-TOP: #b5c4df 1pt solid; BORDER-RIGHT: medium none; PADDING-TOP: 3pt">
<span style="font-weight:bold">From: </span>Alex Williamson <<a href="mailto:alex.l.williamson@gmail.com">alex.l.williamson@gmail.com</a>><br>
<span style="font-weight:bold">Date: </span>Tuesday, December 13, 2016 at 3:23 PM<br>
<span style="font-weight:bold">To: </span>Anjali Kulkarni <<a href="mailto:anjali@juniper.net">anjali@juniper.net</a>><br>
<span style="font-weight:bold">Cc: </span>"<a href="mailto:vfio-users@redhat.com">vfio-users@redhat.com</a>" <<a href="mailto:vfio-users@redhat.com">vfio-users@redhat.com</a>><br>
<span style="font-weight:bold">Subject: </span>Re: [vfio-users] Problem with vfio-pci device type and memory > 10G (or 8G)<br>
</div>
<div><br>
</div>
<div>
<div>
<div dir="ltr">
<div class="gmail_extra">
<div class="gmail_quote">On Tue, Dec 13, 2016 at 4:15 PM, Anjali Kulkarni <span dir="ltr">
<<a href="mailto:anjali@juniper.net" target="_blank">anjali@juniper.net</a>></span> wrote:<br>
<blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex">
<div style="word-wrap:break-word;color:rgb(0,0,0);font-size:14px;font-family:Calibri,sans-serif">
<div>Thanks Alex, I think you got it, there is a lot of memory (251G), but it seems to be mostly all in use:</div>
<div>
<p style="margin:0px;font-size:11px;font-family:Menlo"> vmstat -s</p>
<p style="margin:0px;font-size:11px;font-family:Menlo"> 263843488 K total memory</p>
<p style="margin:0px;font-size:11px;font-family:Menlo"> 254973760 K used memory</p>
<p style="margin:0px;font-size:11px;font-family:Menlo"><br>
</p>
<p style="margin:0px;font-size:11px;font-family:Menlo">Everything looks used up?</p>
</div>
</div>
</blockquote>
<div><br>
</div>
<div>The difference being just around 8.5G, so yeah, unless that used memory can get swapped out you're not going to be able to start a VM with an assigned device with the memory size you're looking for. Without an assigned device the VM memory is not pre-allocated
by default, so the resident memory for the VM might only be 100s of MB. With device assignment, every page of VM memory needs to be allocated and mapped through the IOMMU as a potential DMA target for the assigned device. There is no memory overcommit or
swapping of devices making use of device assignment. You're running into the Out-of-Memory killer, and it's never a pleasant thing. Good luck,</div>
<div><br>
</div>
<div>Alex</div>
</div>
</div>
</div>
</div>
</div>
</span>
</body>
</html>