[Date Prev][Date Next]   [Thread Prev][Thread Next]   [Thread Index] [Date Index] [Author Index]

Re: [libvirt-users] [libvirt] Libvirtd running as root tries to access oneadmin (OpenNebula) NFS mount but throws : error: can’t canonicalize path



On 4/13/2016 10:00 AM, John Ferlan wrote:

On 04/13/2016 09:23 AM, TomK wrote:
On 4/13/2016 1:33 AM, Martin Kletzander wrote:
On Tue, Apr 12, 2016 at 06:24:16PM -0400, TomK wrote:
On 4/12/2016 5:08 PM, John Ferlan wrote:
Having/using a root squash via an NFS pool is "easy" (famous last
words)

Create some pool XML (taking the example I have)

% cat nfs.xml
<pool type='netfs'>
      <name>rootsquash</name>
      <source>
          <host name='localhost'/>
          <dir path='/home/bzs/rootsquash/nfs'/>
          <format type='nfs'/>
      </source>
      <target>
          <path>/tmp/netfs-rootsquash-pool</path>
          <permissions>
              <mode>0755</mode>
              <owner>107</owner>
              <group>107</group>
          </permissions>
      </target>
</pool>

In this case 107:107 is qemu:qemu and I used 'localhost' as the
hostname, but that can be a fqdn or ip-addr to the NFS server.

You've already seen my /etc/exports

virsh pool-define nfs.xml
virsh pool-build rootsquash
virsh pool-start rootsquash
virsh vol-list rootsquash

Now instead of

     <disk type='file' device='disk'>
       <source file='/var/lib/one//datastores/0/38/disk.0'/>
       <target dev='hda'/>
       <driver name='qemu' type='qcow2' cache='none'/>
     </disk>

Something like:

    <disk type='volume' device='disk'>
      <driver name='qemu' type='qemu' cache='none'/>
      <source pool='rootsquash' volume='disk.0'/>
      <target dev='hda'/>
    </disk>

The volume name may be off, but it's perhaps close.  I forget how to do
the readonly bit for a pool (again, my focus is elsewhere).

Of course you'd have to adjust the nfs.xml above to suit your
environment and see what you see/get.  The privileges for the pool and
volumes in the pool become the key to how libvirt decides to "request
access" to the volume.  "disk.1" having read access is probably not an
issue since you seem to be using it as a CDROM; however, "disk.0" is
going to be used for read/write - thus would have to be appropriately
configured...

Thanks John!  Appreciated again.

No worries, handle what's on the plate now and earmark this for checking
once you have some free cycles.  I can temporarily hop on one leg by
using Martin Kletzander's workaround (It's a POC at the moment).

I'll have a look at your instructions further but wanted to find out if
that config nfs.xml is a one time thing correct?  I'm spinning these up
at will via the OpenNebula GUI and if I have update for each VM, that
breaks the Cloud provisioning.  I'll go over your notes again. I'm
optimistic.   :)

The more I'm thinking about it, the more I am convinced that the
workaround is actually not a workaround.  The only thing you need to do
is having execute for others (precisely for 'nobody' on the nfs share)
in the whole path on all directories.  Without that even the pool won't
be usable from libvirt.  However it does not pose any security issue as
it only allows others to check the path.  When qemu is launched, it has
the proper "label", meaning uid:gid to access the file so it will be
able to read/write or whatever permissions you set there.  It's just
that libvirt does some checks that the path exists for example.

Hope that's understandable and it will resolve your issue permanently.

Have a nice day,
Martin


_______________________________________________
libvirt-users mailing list
libvirt-users redhat com
https://www.redhat.com/mailman/listinfo/libvirt-users
The only reason I said that this might be a 'workaround' is due to John
Farlan commenting that he'll look at this later on.  Ideally the
opennebula community keeps the other permissions to nill and presumably
they work on NFSv3 per the forum topic I included earlier from them.
But if setting the permissions on nobody to allow for the functionality,
I would be comfortable with that.

Martin and I were taking different paths...  But yes, it certainly makes
sense given your error message about canonical path and the need for
eXecute permissions... I think I started wondering about that first, but
then jumped into the NFS pool because that's what my reference point is
for root-squash. Since root squash essentially sends root requests as
"nfsnobody" (IOW: others not the user or group), then the "o+x" approach
is the solution if you're going directly at the file.

John

Yes, appears the o+x is the only way right now. It definitely tries to access the share as root though, on CentOS 7 since I also tried to add nfsnobody and nobody to the oneadmin group and that did not work either. Seems OpenNebula doesn't have this issue with NFSv3 running on Ubuntu:

[root mdskvm-p01 ~]# rmdir /tmp/netfs-rootsquash-pool
[root mdskvm-p01 ~]# cat nfs.xml
<pool type='netfs'>
     <name>rootsquash</name>
     <source>
         <host name='opennebula01'/>
         <dir path='/var/lib/one'/>
         <format type='nfs'/>
     </source>
     <target>
         <path>/tmp/netfs-rootsquash-pool</path>
         <permissions>
             <mode>0755</mode>
             <owner>9869</owner>
             <group>9869</group>
         </permissions>
     </target>
</pool>
[root mdskvm-p01 ~]#
[root mdskvm-p01 ~]#

[root mdskvm-p01 ~]# virsh pool-define nfs.xml
Pool rootsquash defined from nfs.xml

[root mdskvm-p01 ~]# virsh pool-build rootsquash
Pool rootsquash built

[root mdskvm-p01 ~]# virsh pool-start rootsquash
error: Failed to start pool rootsquash
error: cannot open path '/tmp/netfs-rootsquash-pool': Permission denied

[root mdskvm-p01 ~]# virsh vol-list rootsquash
error: Failed to list volumes
error: Requested operation is not valid: storage pool 'rootsquash' is not active

[root mdskvm-p01 ~]# ls -altri /tmp/netfs-rootsquash-pool
total 4
     133 drwxrwxrwt. 14 root     root     4096 Apr 14 00:05 ..
68785924 drwxr-xr-x   2 oneadmin oneadmin    6 Apr 14 00:05 .
[root mdskvm-p01 ~]#

[root mdskvm-p01 ~]# id oneadmin
uid=9869(oneadmin) gid=9869(oneadmin) groups=9869(oneadmin),992(libvirt),36(kvm)
[root mdskvm-p01 ~]# id nobody
uid=99(nobody) gid=99(nobody) groups=99(nobody),9869(oneadmin)
[root mdskvm-p01 ~]# id nfsnobody
uid=65534(nfsnobody) gid=65534(nfsnobody) groups=65534(nfsnobody),9869(oneadmin)
[root mdskvm-p01 ~]# id root
uid=0(root) gid=0(root) groups=0(root)
[root mdskvm-p01 ~]#

[root mdskvm-p01 ~]# ps -ef|grep -i libvirtd
root 352 31058 0 00:31 pts/1 00:00:00 grep --color=auto -i libvirtd root 1459 1 0 Apr11 ? 00:07:40 /usr/sbin/libvirtd --listen --config /etc/libvirt/libvirtd.conf
[root mdskvm-p01 ~]#



[root mdskvm-p01 ~]# umount /var/lib/one
[root mdskvm-p01 ~]# mount --no-canonicalize /var/lib/one
[root mdskvm-p01 ~]# umount /var/lib/one
[root mdskvm-p01 ~]# mount /var/lib/one
[root mdskvm-p01 ~]# mount|tail -n 1
192.168.0.70:/var/lib/one on /var/lib/one type nfs4 (rw,relatime,vers=4.0,rsize=8192,wsize=8192,namlen=255,soft,proto=tcp,port=0,timeo=600,retrans=2,sec=sys,clientaddr=192.168.0.60,local_lock=none,addr=192.168.0.70)
[root mdskvm-p01 ~]# umount /var/lib/one
[root mdskvm-p01 ~]# mount --no-canonicalize /var/lib/one
[root mdskvm-p01 ~]# mount|tail -n 1
192.168.0.70:/var/lib/one on /var/lib/one type nfs4 (rw,relatime,vers=4.0,rsize=8192,wsize=8192,namlen=255,soft,proto=tcp,port=0,timeo=600,retrans=2,sec=sys,clientaddr=192.168.0.60,local_lock=none,addr=192.168.0.70)
[root mdskvm-p01 ~]# su - oneadmin
Last login: Thu Apr 14 00:27:59 EDT 2016 on pts/0
[oneadmin mdskvm-p01 ~]$ virsh -d 1 --connect qemu:///system create /var/lib/one//datastores/0/47/deployment.0
create: file(optdata): /var/lib/one//datastores/0/47/deployment.0
error: Failed to create domain from /var/lib/one//datastores/0/47/deployment.0 error: can't canonicalize path '/var/lib/one//datastores/0/47/disk.1': Permission denied
[oneadmin mdskvm-p01 ~]$




CONTROLLER ( NFS Server )

[oneadmin opennebula01 one]$ ls -ld /var{,/lib{,/one{,/datastores{,/0{,/47{,/disk.1}}}}}}
drwxr-xr-x. 19 root     root       4096 Apr  4 21:26 /var
drwxr-xr-x. 28 root     root       4096 Apr 13 03:30 /var/lib
drwxr-x---. 12 oneadmin oneadmin   4096 Apr 14 00:40 /var/lib/one
drwxrwxr-x   6 oneadmin oneadmin     46 Mar 31 02:44 /var/lib/one/datastores
drwxrwxr-x 8 oneadmin oneadmin 60 Apr 13 23:31 /var/lib/one/datastores/0 drwxrwxr-x 2 oneadmin oneadmin 68 Apr 13 23:32 /var/lib/one/datastores/0/47 -rw-r--r-- 1 oneadmin oneadmin 372736 Apr 13 23:32 /var/lib/one/datastores/0/47/disk.1
[oneadmin opennebula01 one]$



NODE ( NFS Client )

[oneadmin mdskvm-p01 ~]$ ls -ld /var{,/lib{,/one{,/datastores{,/0{,/47{,/disk.1}}}}}}
drwxr-xr-x. 21 root     root       4096 Apr 11 07:10 /var
drwxr-xr-x. 45 root     root       4096 Apr 13 04:11 /var/lib
drwxr-x---  12 oneadmin oneadmin   4096 Apr 14 00:39 /var/lib/one
drwxrwxr-x   6 oneadmin oneadmin     46 Mar 31 02:44 /var/lib/one/datastores
drwxrwxr-x 8 oneadmin oneadmin 60 Apr 13 23:31 /var/lib/one/datastores/0 drwxrwxr-x 2 oneadmin oneadmin 68 Apr 13 23:32 /var/lib/one/datastores/0/47 -rw-r--r-- 1 oneadmin oneadmin 372736 Apr 13 23:32 /var/lib/one/datastores/0/47/disk.1
[oneadmin mdskvm-p01 ~]$



Cheers,
Tom K.
-------------------------------------------------------------------------------------
Living on earth is expensive, but it includes a free trip around the sun.


[Date Prev][Date Next]   [Thread Prev][Thread Next]   [Thread Index] [Date Index] [Author Index]