REST service for libvirt to simplify SEV(ES) launch measurement

[Dr. David Alan Gilbert]

* Tobin Feldman-Fitzthum (tobin at linux.ibm.com) wrote:
> 
> 
> On 3/3/22 12:20 PM, Daniel P. Berrangé wrote:
> > On Fri, Feb 25, 2022 at 03:10:35PM -0500, Tobin Feldman-Fitzthum wrote:
> >>
> >>
> >> On 2/24/22 7:26 AM, Daniel P. Berrangé wrote:
> >>> On Wed, Feb 23, 2022 at 03:33:22PM -0500, Tobin Feldman-Fitzthum wrote:
> >>>>
> >>>>
> >>>> On 2/23/22 1:38 PM, Dov Murik wrote:
> >>>>> +cc Tobin, James
> >>>>>
> >>>>> On 23/02/2022 19:28, Daniel P. Berrangé wrote:
> >>>>>> Extending management apps using libvirt to support measured launch of
> >>>>>> QEMU guests with SEV/SEV-ES is unreasonably complicated today, both for
> >>>>>> the guest owner and for the cloud management apps. We have APIs for
> >>>>>> exposing info about the SEV host, the SEV guest, guest measurements
> >>>>>> and secret injections. This is a "bags of bits" solution. We expect
> >>>>>> apps to them turn this into a user facting solution. It is possible
> >>>>>> but we're heading to a place where every cloud mgmt app essentially
> >>>>>> needs to reinvent the same wheel and the guest owner will need to
> >>>>>> learn custom APIs for dealing with SEV/SEV-ES for each cloud mgmt
> >>>>>> app. This is pretty awful.  We need to do a better job at providing
> >>>>>> a solution that is more general purpose IMHO.
> >>>>>>
> >>>
> >>>> Note in particular that we provide a client script called LaunchVM.py
> >>>> that uses libvirt to start an SEV VM in conjunction with the attestation
> >>>> server. This is basically a stand in for a management app or cloud
> >>>> control plane. The modifications needed to launch an SEV VM are not
> >>>> particularly extensive. I agree with some of your comments though. In
> >>>> some ways it might be nice to have libvirt take care of things and hide
> >>>> the complexity from the management app.
> >>>
> >>> LaunchVM.py nicely illustrates my concerns. Every application that
> >>> uses libvirt that knows how to start VMs, now needs to be changed
> >>> to support the series of operations shown in LaunchVM.py. THe guest
> >>> owner probably can't use LaunchVM.py except for demoware, as they'll
> >>> need a equivalent that talks to the API of their cloud mgmt app,
> >>> of which there are many. 
> >>>
> >>>> When we started working on our attestation server, our initial plan was
> >>>> to make PRs to libvirt that would add one end of the attestation API to
> >>>> libvirt, which would directly query the KBS. This is basically what you
> >>>> are proposing. We decided against this for a couple of reasons.
> >>>>
> >>>> First, we were concerned that libvirt might not have network
> >>>> connectivity to an arbitrary attestation server in a cloud environment.
> >>>> We had envisioned that the guest owner would provide a URI for their
> >>>> attestation server as part of the XML. This assumes that the node where
> >>>> the VM is going to run can connect to an attestation server living
> >>>> somewhere on the internet. I think that this might be challenging in
> >>>> some cloud environments. By having the management app connect to libvirt
> >>>> and the attestation server, we add some flexibility.
> >>>
> >>> Agreed, we can't assume that libvirt will always have ability to
> >>> connect to an arbitrary service on the internet.
> >>>
> >>> That said, it does not neccessarily need this ability. If the user
> >>> gives a URL of 'https://myhost.com/attest', the cloud doesn't have
> >>> to give that straight to libvirt. The cloud software could have a
> >>> attestation proxy server. So they could tell libvirt to use the
> >>> URI https://10.0.0.5/attest, and then libvirt connects to that,
> >>> it will proxy the calls through the guest owner's real attestation
> >>> server.
> >>
> >> This might slightly contradict the idea of the management app being out
> >> of the loop, but I guess setting up a proxy isn't very difficult. I
> >> think CSPs already do this kind of thing to enable other features.
> > 
> > The difference I see with a proxy approach is that it ought to end
> > up being a dumb transport. It won't have to define any protocol or
> > interpret the data, just blindly pass data back & forth.
> > 
> > This is already something often done with VNC where the user connects
> > to a public endpoint exposed by the cloud on its internet boundary,
> > which then forwards the data onto QEMU's real VNC server on the
> > compute host. In the VNC case, the public facing side often does
> > websockets encapsulation, while the private side is pure VNC, but
> > it still doesn't ened to understnd the VNC protocol so it is fairly
> > easy to setup such a proxy.
> > 
> > A proxy could also address the other problem I've realized. At least
> > the first VM to be booted on a given cloud might be harder to attest
> > because the attestation service would need to exist outside the cloud
> > being used because the guest owner won't trust anything initially.
> > 
> > This could imply that the attestation service is on a local machine
> > controlled by the guest owner, even their local laptop. The implication
> > is that the attestation service could easily be stuck behind NAT and
> > be unable to accept incoming connections from the cloud.
> > 
> > To address this the proxy might need to sit in the middle and accept
> > incoming connections from both the guest owner's attestation service
> > and from libvirt on the compute host.
> > 
> > IOW, with VNC you get a unidirectional flow of connection
> > establishment
> > 
> >   guest owner -> cloud proxy -> compute host
> > 
> > but with the attestation service you might need a flow
> > with the second arrow reversed ie
> > 
> >   guest owner -> cloud proxy <- compute host
> > 
> > 
> Yeah I think a proxy is a reasonable solution. I am still not sure where
> people are going to run their attestation servers in practice. We have
> been assuming that they could be anywhere. I don't know if people will
> insist on running them locally or if they will decide to trust some
> managed attestation solution. I guess we will see.
> 
> > Once the first VM is bootstrapped, it could be used to run an
> > attestation service that is local to the cloud, avoiding the
> > need for the traffic to traverse the internet when booting
> > future VMs, and removing the dependancy on the guest owner having
> > a local machine to run an attestation service on.
> > 
> > 
> >>> The needs of all the apps look similar enough, because they are all
> >>> ultimately constrained by the functionality made available by SEV(ES).
> >>> Thus at least wrt apps doing traditional VM management using QEMU,
> >>> it feels like it should be practical to come up with a common solution.
> >>> I am all for common ground here. I had basically given up on it, but
> >> maybe libvirt has enough influence to set a standard. In your first
> >> email you said that there is a relatively short window to come up with
> >> something and I think that is probably correct.
> > 
> > Yes, there are a handful of major cloud software projects that use
> > libvirt, and once they've all built their own solution, it won't be
> > so easy to convince them to change into a shared solution, as it
> > would create back compat problems for them.
> > 
> >>> I can understand if it is harder to achieve commonality with tech
> >>> like libkrun though, since that's consuming virt in quite a different
> >>> way at the userspace level.
> >> Yeah, extending the focus beyond SEV(-ES) with QEMU might make things
> >> more difficult. There is some discussion right now about trying to find
> >> common ground between SEV-SNP and TDX attestation, but I assume that is
> >> all out of scope since libvirt isn't really involved.
> > 
> > I admit I don't know much about TDX, but from what I've understood
> > talking to other people, SEV-SNP might not end up looking all that
> > different. IIUC the attestation has to be initiated from inside
> > the SNP guest after CPUs are running. It is going need to be run
> > as early as possible and while you might be able todo it in the
> > initrd, it feels likely that it could be put into the firmware (OVMF)
> > instead, such that it does the validation before even loading the
> > kernel. This would facilitate supporting it with arbitrary guest OS,
> > as the firmware is common to all. We can't assume the firmware will
> > have direct network connectivity to any attestation service needed
> > to verify the boot. This implies the firmware might need to talk
> > to the host via something like virtio-serial / virtio-vsock, from
> > where libvirt or QEMU can proxy the traffic onto the real attestation
> > service.  Such an architecture might end up aligning quite well with
> > SEV/SEV-ES, possible allowing the same protocol to be used in both
> > cases, just with differnt ultimate end points (libvirt for SEV(-ES)
> > vs guest firmware for SEV-SNP).
> 
> Yeah that is an interesting point. Most SNP approaches that I have seen
> so far use the kernel/initrd to handle decryption. There is potentially
> a gap if the kernel/initrd are not themselves part of the measurement
> that is provided in the attestation report. We have been using this
> measured direct boot thing for SEV(-ES) and I think it can be extended
> to SEV-SNP as well. This would close that gap and make it feasible to do
> the decryption in the kernel.

With the direct boot setup, it feels like using 'clevis' in the initrd
would be the right way to wire things to disk decryption.
[ https://github.com/latchset/clevis ]
It would need a 'pin' writing for SNP that then did whatever
communication mechanism we settled on.

(A clevis pin might also be the way to wire the simple disk key from
your EFI/SEV mechanism up to LUKS? )

Dave

> There might be reasons to do the measurement earlier, however. For
> instance, it is easier to keep track of the hashes of fewer things (just
> the firmware vs the firmware + initrd + kernel + cmdline). As you say
> networking becomes a bit of an issue if you do the attestation in
> firmware. Using a local device that is handled by libvirt could be a
> good solution.
> 
> I'm sure we could come up with a protocol that is general enough to
> handle both pre-attestation and runtime attestation, but there
> definitely are some differences. For instance with pre-attestation we
> know that there will basically be two calls from management app to
> attestation server. This is defined by the way that we inject secrets.
> With SNP things are much more flexible. We can setup a persistent
> connection between the guest and the attestation server and ask for
> secrets throughout the runtime of the guest. It might take some thinking
> to reconcile these approaches and could put some dependencies on how the
> guest is supposed to behave, which isn't really our business (although a
> firmware-based solution could be reasonable).
> 
> -Tobin
> > 
> > Regards,
> > Daniel
> 
-- 
Dr. David Alan Gilbert / dgilbert at redhat.com / Manchester, UK