[libvirt] Redesigning Libvirt: Adopting use of a safe language

[Daniel P. Berrange]

On Thu, Nov 16, 2017 at 04:55:55PM -0500, John Ferlan wrote:
> On 11/14/2017 12:27 PM, Daniel P. Berrange wrote:
> > When libvirt was created, C was the only viable choice for anything aiming to be
> > a core system library component. At that time 2005, aside from C there were
> > common choices of Java, Python, Perl. Java was way too heavy for a low level
> > system component, Python was becoming popular but not widely used for low level
> > system services and Perl was on a downward trend. None of them are accessible to
> > arbitrary languages as libraries, without providing a RPC based API service. As
> > it turns out libvirt did end up having RPC based approach for many virt drivers,
> > but the original approach was to be a pure library component.
> > 
> > IOW it is understandable why C was chosen back in 2005, but 12 years on the world
> > around us has changed significantly. It has long been accepted that C is a very
> > challenging language to write "safe" applications. By "safe" I mean avoiding the
> > many problems that lead to critical security bugs. In particular the lack of a
> > safe memory management framework leads to memory leaks, double free's, stack or
> > heap corruption and more. The lack of strict type safety just compounds these
> > problems. We've got many tools to help us in this area, and at times have tried
> > to design our APIs to avoid problems, but there's no getting away from fact that
> > even the best programmers will continually screw up memory management leading to
> > crashes & security flaws. It is just a fact of life when using C, particularly if
> > you want to be fast at accepting new feature proposals.
> > 
> > It is no surprise that there have been no new mainstream programming languages in
> > years (decades) which provide an inherantly unsafe memory management framework.
> > Even back in 2005 security was a serious challenge, but in the last 10+ years
> > the situation has only got worse with countless high profile security bugs a
> > direct result of the choice to use C. Given the threat's faced today, one has to
> > seriously consider the wisdom of writing any new system software in C. In another
> > 10 years time, it would not surprise me if any system software still using C is
> > considered an obsolete relic, and ripe for a rewrite in a memory safe language.
> > 
> 
> Programming languages come and (well) go - it just so happens the C has
> been a survivor. There's always been challengers promising something,
> but eventually falling by the wayside when either they fail to deliver
> their promise or the next sexy language comes along. In my 30 years
> (eeks) even with all warts C has been there. It was certainly better
> than writing in assembly/macro or BLISS. I recall converting a lot of
> BLISS to C when I first started.

I had to go an look up what BLISS was - that's a new (well old) one for
me :-)

Just thinking about the languages that have risen up while I've been
using C, I'm not surprised that C has been a survivor in the area that
libvirt live. There's been a plethora of dynamic and/or scripting
languages that have become popular (perl, python, ruby, javascript,
to name but 4). None of these have really been a credible choice for
usage libvirt. Use of interpretors by default has limited their perf,
and forces a multi-process model to get any true parallelism. The
latter has been a huge bottleneck for OpenStack Nova's concurrency.
There's then the elephant in the room, Java, and with its JVM model
the memory footprint of running it is just crazy. In all of them,
interfacing to C is possible, but horribly unpleasant work to a large
degree. And of course there's always C++, but that takes C's already
complex world and makes it even more complex & leaves all the dangerous
aspects of C still there, so when you shoot yourself in the foot, it
blows away your entire leg. There's quite a few other interesting
languages around, but none of them have received such mainstream
usage, so are hard to justify if you want a broad contributor set.

Rust & Go by comparison, have offered something pretty unique in
languages that are still compiled to native and fairly low level
and easy to interface with C. Between them they have strong potential
to eliminate the need to use C for the majority of remaining usage
scenarios, which can't be said of other languages I've described
above. 

> There is something to be said about the "devil you know" vs. the one you
> don't! Just as much as there is a need to keep yourself "current" with
> technology trends. The latter becomes harder to do the longer I do this.

I don't disagree, but I think you would be pleasantly surprised at how
easy it is to learn Go if coming from a C background (or indeed a Python
or Java background too).

> 
> > There are long term implications for the potential pool of contributors in the
> > future. There has always been a limited pool of programmers able todo a good job
> > in C, compared to those who know higher level languages like Python/Java. A
> > programmer write bad code in any language, but in C/C++ that bad code quickly
> > turns into a serious problem. Libvirt has done ok despite this, but I feel our
> > level of contribution, particularly "drive by" patch submissions, is held back
> > by use of C. Move forward another 10 years, and while C will certainly exist, I
> > struggle to imagine the talent pool being larger. On the contrary I would expect
> > it to shrink, certainly in relative terms, and possibly in absolute terms, as
> > other new languages take C's place for low level systems programming. 10 years
> > ago, Docker would have been written in C, but they took the sensible decision to
> > pick Go instead. This is happening everywhere I look, and if not Go, then Rust.
> 
> I'm not convinced that "drive by" patch submissions are those we seek.
> As stated, libvirt is a fairly complex project. I would think drive by
> submissions lead to more problems regardless of the language chosen
> because a reviewer spends so much of his/her valuable time trying to
> assist the new contributor only to eventually learn that it is a drive
> by. Then those that are committed to the project are left to decide to
> drop the drive by submission or support it for years to come. Invariably
> there's some integration interaction that was missed.
> 
> I would hope our long term goal would be build up not only contributors,
> but more importantly reviewers. Again, doesn't matter what language is
> chosen, since libvirt has review requirements then it needs reviewers.
> If GO is a language from which to draw new contributors and more
> importantly reviewers, then great.

Yep, I don't disagree about our need for reviewers, as much as we need
contributors, if not more. I think pretty much every non-trivial project
I have seen suffers from a need for more reviewers. Every time you make
reviewers more efficient, you enable a greater flow of patches, and so
you enable more contributions. Just like building roads to solve traffic
jams, never actually solves traffic jams. So I won't claim that choice of
language is the driving factor in availablity of reviewers.  What I will
say is that more productive language would lets us focus attention on
the more interesting problems to libvirt, instead of working on general
infrastructure, portaibility and all the things that language doesn't
provide us. IOW reviewers would still be horribly overloaded, but the
stuff they would be reviewing would be more useful.

> Maybe it's a bit of 'bias' and terminology, but I've always thought
> there is a difference between programmer and software engineer. My FUD
> is that we attract too many of the former and not enough of the latter
> that are necessary to solve that complex issue.

I think that is basically true of every single open source software
project that's out there.


> > The other big trend of the past 10 years has been the increase in CPU core
> > counts. My first libvirt dev machine had 1 physical CPU with no cores or threads
> > or NUMA. My current libvirt dev machine has 2 CPUs, each with 6 cores, for 12
> > logical CPUs. Common server machines have 32/64 logical CPUs, and high end has
> > 100's of CPUs. In 10 years, we'll see high end machines with 1000's of CPUs and
> > entry level with mere 100's. IOW good concurrency is going to be key for any
> > scalable application. Libvirt is actually doing reasonably well in this respect
> > via our heavily threaded libvirtd daemon. It is not without cost though with
> > ever more complex threading & locking models, which still have scalability
> > problems. Part of the problem is that, despite Linux having very low overhead
> > thread spawning, threads still consume non-trivial resources, so we try to
> > constrain how many we use, which forces an M:N relationship between jobs we need
> > to process and threads we have available.
> > 
> 
> So GO's process/thread model is then lightweight?  What did they learn
> that the rest of us ought to know! Or is this just a continuation of the
> libvirtd discussion?

So in C you officially have the option of pthreads, which we use heavily.
As you know a pthread maps 1:1 to an operating system thread (at least in
the Linux impl, IIUC, that's not technically required by POSIX specs).
Each thread has a fixed stack size associated with it (defaults to 8MB
in size on Fedora at least). Of course a thread only uses 8MB of physical
memory if it actually touches all those pages. If you exceed the stack
bad things happen, so it is hard to pick a safe smaller size for pthread
stacks. The OS schedules the threads and deals with context switching as
for any normal process. Linux pthreads is nicely efficient compared to
the original LinuxThreads impl and other OS like Solaris, but it is still
a fairly heavy context switch.

Alternatively you have the option of inventing a Coroutine concept like
QEMU has used in its block layer. That lets a single OS thread run multiple
userspace threads, typically the application will switch between coroutines
manually at key points (like I/O operations). Coroutine context switching
is lighter than thread switching so it can be beneficial. You would typically
use a smaller stack size, but it is still a fixed stack, so bad stuff happens
if you pick too small a size. You have to decide which OS level thread runs
which coroutines manually.

Goroutines are basically a union of the thread + coroutine concepts. The
Go runtime will create N OS level threads, where the default N currently
matches the number of logical CPU cores you host has (but is tunable to
other values). The application code just always creates Goroutines which
are userspace threads just like coroutines. The Go runtime will dynamically
switch goroutines at key points, and automatically pick suitable OS level
threads to run them on to maximize concurrency. Most cleverly goroutines
have a 2 KB default stack size, and runtime will dynamically grow the
stack if that limit is reached. So this avoids the problem of picking a
suitable stack size, and avoids any danger of overruns. As a result it
is possible for a process to create *1000's* of goroutines and have less
overhead than if you tried todo the same thing in C with threads+coroutines
manually.

The fact that goroutines are so cheap means you can use simpler threading
designs for applications. eg instead of the approach where libvirt tries
to multiplex all I/O into a single event thread, and then have a pool of
threads for RPC calls, but then add another pool for threads for RPC calls
that must always run quickly, we could dramatically simplify things. Normal
practice in Go would just have a single Goroutine for each client socket
connection, and this would spawn a single Goroutine for each RPC call that
needs to be run. Essentially eliminates all the throttling & queuing of
calls that we do, which removes the bottlenecks it inherantly creates.
cf the problems that Prerna is trying to solve where our main loop is
getting block by QEMU event handling and the increasingly complex solutions
we're trying to invent to deal with it.


> Still it seems the pendulum has swung back to hardware and software
> needs to catch up. It used to be quantum leaps in processor speed as it
> related to chip size/density - now it's just leaps in the ability to
> partition/thread at the chip level. I'd hate to tell you about the boat
> anchor I had on my desktop when I first started!

IBM solved everything in the 60's & 70's on the mainframe, and we're still
trying to reinvent all the solutions they had :-P

> > Two fairly recent languages, Go & Rust, have introduced new credible options for
> > writing systems applications without sacrificing the performance of C, while
> > achieving the kind of ease of use / speed of development seen with languages
> > like Python. It goes without saying that both of them are memory safe languages,
> > immediately solving the biggest risk of using C / C++.
> > 
> 
> If memory mgmt and security flaws are the driving force to convert to
> GO, then can it be claimed unequivocally that GO will be the panacea to
> solve all those problems? Even the best intentions don't always work out
> the best. If as pointed out in someone else's response there have been
> CVE's from/for GO centric apps - how many of those are GO related and
> how many are App related? Not that it matters, but the point is we're
> shifting some amount of risk for timely fixes elsewhere and shifting the
> backwards compatible story elsewhere which could be the most
> problematic. Not everyone has the same end goal for ABI/API
> compatibility. Add to that the complexity of ensuring that a specific
> version of some package you've based your product/reputation on.

NB, I'm certainly not claiming it will solve all security flaws. Far
from it, there's been plenty of screwups we've done that are not at all
related to choice of language. I am claiming that we'll eliminate all
those flaws related to use of unsafe memory mangement. ie buffer overflows,
double frees, use of free'd memory, and all the other fun ways in which
we screw up and crash libvirtd, often enabling security attacks (even
if we don't file CVEs for most of them)

> Curious, is the performance rated vs. libc memory alloc/free or
> something else? I don't recall ever being on a project that didn't have
> some sort of way to "rewrite" the memory mgmt code. Whether it was shims
> to handle project specific needs or usage of caches to avoid the awful
> *alloc/free performance. Doing the GC is great, but what is the cost.
> Perhaps something we don't know until we got further down that path.

I don't see a compelling case where libvirt has performance critical
memory management requirements. Indeed use of manual malloc/free is
far from offering the best performance from a memory mgmt POV in
general. Our biggest performance problems come when we inherantly
self-limit our performance by using fewer threads than are really
needed to deal with the number of VMs we're managing. Assuming the GC
is not totally useless, I don't see a reason why it is an issue for
libvirt. In terms of scope Docker is very similar to what libvirt
tries todo, and probably has greater performance requirements than
libvirt because container density on a machine is usually much
higher than VM density. Beyond that you have apps like Etcd and
Kubernetes which have an order of magnitude greater performance
needs than libvirt, as they're managing across enter clusters of
100's of machines or more.  IOW, use of GC is not a concern for
me, and clearly outweighs the downsides of our current manual
approach both in terms of code simplicity and reliabity.

> > The particularly interesting & relevant innovation of Go is the concept of
> > Goroutines for concurrent programming, which provide a hybrid kernel/userspace
> > threading model. This lowers the overhead of concurrency to the point where you
> > can consider spawning a new goroutine for each logical job. For example, instead
> > of having a single thread or limited pool of threads servicing all QEMU monitor
> > sockets & API clients, can you afford to have a new goroutine dedicated to each
> > monitor socket and API client. That has the potential to dramatically simplify
> > use of concurrency while at the same time allowing the code to make even better
> > use of CPUs with massive core counts.
> 
> Sounds promising and complicated, but is the risk of libvirt discovering
> some flaw or limitation in goroutine's worth it?  IOW: Would libvirt be
> blazing a new trail or are other consumers that have "helped" work
> through the initial issues.

We would not be anywhere near pushing the boundaries of Go. Apps like
Etcd / Kubernetes stretch it far more than we would.

> Oh, and license wise it would seem we'd have to be careful, true? At
> least w/r/t attempting to utilize packages written or listed on the wiki
> page link. From just a quick scan there, it seems to be numerous
> "packages" available and some list difference licenses.

Yes, licensing is always a concern. Most commonly I see Go code under
more permissive licenses (BSD, Apache) than libvirt has traditionally
used. L(GPL)v2+ is compatible with both BSD & Apache, though for Apache
it relies on v2+ becoming v3+. L(GPL)v2-only code is a problem with
Apache compatibilty. We do unfortunately suffer from GPLv2-only in the
VirtualBox driver, inherited from the VirtualBox XPCOM API which is
GPLv2-only and hence why we had to move it out of libvirt.so into
libvirtd despite it being a stateless driver. The only real good option
there, aside from deleting it, is to isolate the VirtualBox driver still
further in a standalone process such that its problems are confined.
The refactoring of libvirtd I suggested would probably help with the latter

> Also, once chosen what happens if/when issues or incompatibilities are
> discovered in some package? Do we follow the same principle of GNULIB
> and try to fix it ourselves or somehow work around it? As I've learned
> through time - "how" someone else fixes a problem may not work out best
> and the degree of importance of the problem can result in delays in
> getting a resolution. Having some amount of control is nice and we just
> have to weigh the risk(s) of giving some of that away.

I don't have a clear answer for this, since it would probably depend on
the kind of problems we hit.  This kind of unknown is why a cautious
approach would be best, starting at the edge of libvirtd where scope
for interactions and/or impact on other work is limited. eg virtlogd
and virtlockd are both very self-contained services, so could be good
guinea pigs for initial prooving work without disrupting libvirt in
general.

> > I don't believe that the unique features of Rust, over Go, are important to the
> > needs of libvirt. eg while for QEMU it would be critical to not have a GC
> > doing asynchronous memory deallocation, this is not at all important to libvirt.
> > In fact precisely the opposite, libvirt would benefit much more from having GC
> > take care of deallocation, letting developers focus attention other areas. In
> > general, as from having a memory safe language, what libvirt would most benefit
> > from is productivity gains & ease of contribution. This is the core competancy
> > of Go, and why it is the right choice for usage in libvirt.
> 
> Depends on the GC, right? Is GC context/scope based? or overall APP
> based? There are certainly some particularly hairy uses of memory and
> arguments in libvirt code.

GC approach in Go is mark+swweep, but as mentioned above, when compared to
the scale of other apps using Go, I'm not concerned from libvirt POV.


> > The best way to start, however, is probably to focus on a simple self-contained
> > area of libvirt code. Specifically attack the virtlockd, and/or virtlogd daemons,
> > converting them to use Go. This still need not be done in a "big bang". A first
> > phase would be to develop the server side framework for handling our RPC protocol
> > deserialization. This could then just dispatch RPC calls to the existing C impls.
> > As a second phase, the RPC method impls would be converted to Go. Both of these
> > daemons are small enough that the conversion would be possible across the time
> > of a couple of releases. The hardest bit is likely ensuring compatibility for
> > the re-exec() upgrade model they support, but this is none the less doable.
> > The lessons learned in this would go a long way towards informing the best way
> > to tackle the bigger task of the monolithic libvirtd (or equivalently the swarm
> > of daemons the previous proposal suggests)
> > 
> 
> It will take though "someone" who knows GO and libvirt well enough
> start. At this time, I submit that pool of talent is quite limited. Not
> necessarily GO contributors, but those that understand the libvirt build
> system, how to mash things together, how to write good GO code, and what
> types of considerations one has to make when developing at the OS,
> daemon, and library level.

You can probably guess that the "someone" would be me ;-) You certainly
have a valid point here, which is again my a cautious approach is needed
rather than attempting to much of a "big bang". I certainly would not
start anywhere near the QEMU driver, since chance of disrupting other
devs productivity is far too high.

Regards,
Daniel
-- 
|: https://berrange.com      -o-    https://www.flickr.com/photos/dberrange :|
|: https://libvirt.org         -o-            https://fstop138.berrange.com :|
|: https://entangle-photo.org    -o-    https://www.instagram.com/dberrange :|