[libvirt-users] virsh not detecting hugepage mount; disabled by config?

Wed Dec 14 13:58:48 UTC 2016

-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA256

Thanks a lot you two,

yes hugetlbfs is not mounted, when libvirtd is started… It was a long
night. Did put it to fstab and since I do not allocate them on regular
boots, it won’t eat my ram (that was the main consideration for doing it
like this).
As a last means before going to sleep, I added
    <qemu:arg value='-numa'/>
    <qemu:arg value='node,memdev=mem42'/>
    <qemu:arg value='-object'/>
    <qemu:arg
value='memory-backend-file,size=6G,mem-path=/var/lib/hugetlbfs,share=on,id=mem42'/>
to the xml which worked too (surprise), but I have it working now in the
intended way.
Then I never got to the last part of the qemu.conf option comment so I
had to do another 3 reboots. Well, reading helps.
So the fstab entry reads now:
hugetlbfs   /var/lib/hugetlbfs/libvirt/qemu  hugetlbfs
relatime,rw,pagesize=1073741824  0 0
with /var/lib/hugetlbfs being the mount point in qemu.conf.
At least I didn’t forget to create the mount point before rebooting.

Best regards,
Manuel
> Hi, > > I use gentoo with desktop+systemd profile to run Windows 10 VM with
pci > passthrough and hugetlbs. > > Things I had to do for hugetlbs: >
1) Made sure their support was enabled in kernel > 2) Passed parameters
for their allocation to kernel. There are other > ways, but I use 1GB
page size and I have to allocate them before memory > gets fragmented.
Note, you do that with hugepages=3072. > 3) Passed hugetlbs settings to
libvirt VM domain (XML). > > That is it. I don't mount them myself. I
think (can't check now) they > are automounted to /dev/hugetlbs. By
default, libvirtd runs as root, and > qemu as user qemu. Consuming
hugetlb and releasing them should be > transparent. > > I suggest
starting without softlevel=qemuvm into default runlevel, and > then
check /proc/meminfo. Hugetlbs total and free should be > 0. Do the >
same with qemuvm runlevel. Maybe this runlevel misses something. > > On
2016-12-14 00:05, Manuel Ullmann wrote: >> Hi, >> >> I’m struggling with
virsh not detecting my hugepage mount. >> >> I have the following kernel
command line: >> BOOT_IMAGE=/vmlinuz-4.8.13-gentoo
root=/dev/mapper/gensd-gentoo ro quiet >> splash intel_iommu=on
video=efifb:off,vesafb:off,simplefb:off >>
splash=verbose,theme:livedvd-aurora kvm.ignore_msrs=1 >>
transparent_hugepage=never hugepages=3072 softlevel=qemuvm >> >> My
startup script outputs the following: >> hugetlbfs /var/lib/hugetlbfs
hugetlbfs >> rw,relatime,pagesize=2097152,uid=77,gid=77,mode=0770 0 0 >>
[2016.12.13 22:22:50 virsh 2808] ERROR Failed to start domain win10 >>
[2016.12.13 22:22:50 virsh 2808] ERROR internal error: hugetlbfs >>
filesystem is not mounted or disabled by administrator config >> virsh
was unsuccessful >> >> So hugetlbfs is definitively mounted, but virsh
does either not detect >> it or some magical option, that I’ve not found
in a day searching for >> it, disables it (the documentation could
explain that better). Since the >> Ubuntu guys refer to the
KVM_HUGEPAGES environment, I tried adding it to >> start-stop-daemon
environment, but it seems non-relevant. 77 is the qemu >> user id, but
I’m quite sure the permission issues were excluded (tried >> root
permissions as well). Besides I had permission errors occasionally >>
occasionally and don’t have them anymore. If you could provide a hint to
>> the magical administrator configuration option, that would be
helpful. >> Tried hugeadm as well, pointing the config to the correct
destination. >> >> Thanks in advance, >> Manuel >> >> PS: >> The virsh
script reads like this: >> #!/bin/sh >> cmdline="$(cat /proc/cmdline)"
>> if [[ "${cmdline##* }" == "softlevel=qemuvm" ]]; then >>     mount -o
rw,relatime,pagesize=2097152,uid=77,gid=77,mode=0770 -t >> hugetlbfs
hugetlbfs /var/lib/hugetlbfs >>     sysctl kernel.shmmax=6442450944
>>     cat /proc/mounts | grep hugetlb >> /var/log/virsh.log >>     sed
-i -e '/^hugetlb/{s/^/\#/}' /etc/libvirt/qemu.conf >>     counter=0
>>     while [ "${counter}" -lt 10 && ! -S /var/run/libvirt/libvirt-sock
]; do >>     sleep 1 >>     counter=$(( ++counter )) >>     done >>    
if [ $counter -gt 9 ]; then >>     echo "libvirtd socket generation
timed out" >> /var/log/virsh.log >>     fi >>     if pidof libvirtd
&>/dev/null; then >>     LC_ALL=C /usr/bin/virsh -l /var/log/virsh.log
start win10 >>     if [ $? -gt 0 ]; then >>         echo "virsh was
unsuccessful" >> /var/log/virsh.log >>         reboot >>     fi >>    
else >>     echo "libvirtd is not started" >> /var/log/virsh.log >>    
reboot >>     fi >> fi >> >> reboot is more convenient, since this is a
vfio-igd passthrough vm. >> >> >>
_______________________________________________ >> libvirt-users mailing
list >> libvirt-users at redhat.com >>
https://www.redhat.com/mailman/listinfo/libvirt-users >>

-----BEGIN PGP SIGNATURE-----

iQQcBAEBCAAGBQJYUVAKAAoJELgK9xQKkpX6sTof/2fdLX9qjiNgecKEIzHKEjBY
ln6o7E0nMY796Jf3HQ/v620uVZS3Nf4dRSFC9NVetzAgVcQizwko0ikVDsaCZlNP
5HojcRm2SK8c5CgU/ZLdLPp73PgrORtKYytYldhPQFebgOpNCJGzWAPXGnoLK78s
aHsKGKb5R79+K2Bq5DAQGR5N9wiV84Oi+afNjBunjj1dT+80YPAAtXuRKkDdD3lQ
hVBEdzgRLWW48FdEhIp2TlTJCyx4OwMGnIXaOPXrQSyMW6Omn49ZXYqwIgsOW45S
bzFZ4BPuqhrANne4YDPjuxwtL65YUKAyHZWl/bTBmqkFrpjecxYDnlg219t2KsiM
k3W9GQ0CWR74f0lJLULaV+ome13ydga5pDRe6+dyi7g0O7/4DTPuxQTRo07UiA/i
dQmvtzTAbGdSpGqyFBQqIFGMG4sB8kypXp1PIK06jX1Muon6WCdzAzt1IztLeuSW
qUj/srdnD9rLWWPlIV2NCZ3E5Da4szRD8gbhNIG2hg+Ss50pLdIFwlUbfksryY7B
B6L6KmpOw4LrpCZVFBeFhG5lFns9YSvYNKvSBRwJaxl838j8Kr8B1bo51nMYERL1
wJXK9IyFqx8jkbmFvfwXij5lx3DFBrLEQwymGmaLmJ4FQmnA9491L2F3K+NyG8kX
Yi/w+Xh9ZARoRy8/2SmVYwvB1vZkFpf04adocHxIYMT1P3ZLpCktu3P/DMdkdR26
DcnQedl7qbf7NeldxMiY0zTAti85VFuQggFQlRrB36pRdOXvFMk3mNLBiifICNvP
32sTl/9HnmscWCTSebmK/KjQjrLf9ijbc3V942WziwkEyDbOhfTUn0JfcZJl0FdM
OQ0j00t5wv14KFPP/hpWqRiypwXpboHLpB4k718NZElObUOob0ziz6PJc5iCzJDW
AkNbO1sEDgWN/pPorAYKPtre3TV6qk/k6A13zgNmMdZA4ZHRq6dsZLzCLnEaLwJE
G5WFveDyi4COVpelysArTPrw6aL4LAzIhkRN5A5YCANgsPtnhjql6XmL3sTSN93Z
R42klgPKNZVq5Bkdp2EpE70X+F/4j2jhRNznVp3YDU+6q6aNWHdqw7jCAhEji0aA
49gR3tIUgXp1883zc0vSFIMnCwYQQoCjqGpVZXWLtbxAmJrz5c4XeUCnpplfJW3Z
jRS19MEy6HgKmdkq1a4bRG5XFc87Ga2JWqwSXpGb1EsMvyMU3CQq47D98WqaeC9p
JTOuLMpYvlH+EEbEkL/deysLGVZ7QDTr5P4L+wOzN9jhSOrnIoNlVU5DVP7dhT2L
z1d3UY3cFhoVFLKyfbLF+flK7Y5aygQ/eO/+MlXmd9TI+2qTVehNSXiYTEnnsO0=
=1DU1
-----END PGP SIGNATURE-----