[libvirt] Redesigning Libvirt: Adopting use of a safe language

Wed Nov 15 14:59:20 UTC 2017

On Wed, Nov 15, 2017 at 12:28:30PM +0100, Bjoern Walk wrote:
> Daniel P. Berrange <berrange at redhat.com> [2017-11-14, 05:27PM +0000]:
> > The Problem(s)
> > ==============
> > 
> > When libvirt was created, C was the only viable choice for anything aiming to be
> > a core system library component. At that time 2005, aside from C there were
> > common choices of Java, Python, Perl. Java was way too heavy for a low level
> > system component, Python was becoming popular but not widely used for low level
> > system services and Perl was on a downward trend. None of them are accessible to
> > arbitrary languages as libraries, without providing a RPC based API service. As
> > it turns out libvirt did end up having RPC based approach for many virt drivers,
> > but the original approach was to be a pure library component.
> > 
> > IOW it is understandable why C was chosen back in 2005, but 12 years on the world
> > around us has changed significantly. It has long been accepted that C is a very
> > challenging language to write "safe" applications. By "safe" I mean avoiding the
> > many problems that lead to critical security bugs. In particular the lack of a
> > safe memory management framework leads to memory leaks, double free's, stack or
> > heap corruption and more. The lack of strict type safety just compounds these
> > problems. We've got many tools to help us in this area, and at times have tried
> > to design our APIs to avoid problems, but there's no getting away from fact that
> > even the best programmers will continually screw up memory management leading to
> > crashes & security flaws. It is just a fact of life when using C, particularly if
> > you want to be fast at accepting new feature proposals.
> > 
> > It is no surprise that there have been no new mainstream programming languages in
> > years (decades) which provide an inherantly unsafe memory management framework.
> > Even back in 2005 security was a serious challenge, but in the last 10+ years
> > the situation has only got worse with countless high profile security bugs a
> > direct result of the choice to use C. Given the threat's faced today, one has to
> > seriously consider the wisdom of writing any new system software in C.
> 
> I agree for newly written software. There is almost no reasoning to use
> C for starting another project. Especially given the amount of different
> options of problem-specific languages out there nowadays. But I don't
> think argument holds for existing projects. I would suggest that the
> amount of time that has already been spend in finding and mitigating
> critical security bugs outweighs the possible inherent safety of any new
> language.

I don't think that is the case unless the code is essentially feature
complete and not writing significant new code. That is certainly not
the case with libvirt, which shows no sign of slowing down in terms
of features we must develop. As long as you are continuing to write
non-trivial C code, you are continuing to introduce security bugs.
Further, things that in the past were not considered security flaws,
and increasingly be classed as security flaws.  What has improved is
that we use more tools to detect the security flaws & crashing bugs
after we've introduced them. eg coverity tells us about many screw
ups. It is none the less an ongoing issue.

> > There are long term implications for the potential pool of contributors in the
> > future. There has always been a limited pool of programmers able todo a good job
> > in C, compared to those who know higher level languages like Python/Java. A
> > programmer write bad code in any language, but in C/C++ that bad code quickly
> > turns into a serious problem. Libvirt has done ok despite this, but I feel our
> > level of contribution, particularly "drive by" patch submissions, is held back
> > by use of C. Move forward another 10 years, and while C will certainly exist, I
> > struggle to imagine the talent pool being larger. On the contrary I would expect
> > it to shrink, certainly in relative terms, and possibly in absolute terms, as
> > other new languages take C's place for low level systems programming. 10 years
> > ago, Docker would have been written in C, but they took the sensible decision to
> > pick Go instead. This is happening everywhere I look, and if not Go, then Rust.
> 
> Out of interest, I took a look at the CVE history of both libvirt and
> docker:
> 
>     https://www.cvedetails.com/product/20594/Redhat-Libvirt.html?vendor_id=25
>     https://www.cvedetails.com/product/28125/Docker-Docker.html?vendor_id=13534
> 
> Not sure, how up to date and complete this list is, but for the sake of
> arguments, let's take it. Docker since its creation in 2014 had 15 CVEs,
> 2 of them code execution and 3 of them privilege escalation. On the
> other hand, libvirt had, in the same time frame since 2014, a total of
> 20 CVEs, 1 of them code execution and 2 privilege escalations. The year
> 2014 was even an outlier with 13 CVEs that year. So honestly, in terms
> of security, I don't see a prevailing argument for Go as the better
> language compared to C. Mind as well that the size of the codebase of
> libvirt is somewhat 3-6 times larger then that of docker, depending on
> how you count it.

I don't think that this is a sensible comparison. For libvirt we have
effectively given up and said that a large portion of our APIs have
semantic privileges equivalent to a root shell. This turn means that
when libvirtd crashes, we claim that it is not a security flaw. We have
many 100's of crashes we've solved, and their frequency of occurrance
is not slowing down. IOW, if we actually classified all our crashes as
CVEs we would have orders of magnitude more CVEs. Essentially we are
ignoring the scale of the problem.

> > We push up against the boundaries of what's sane todo in C in other ways too.
> > For portability across operating systems, we have to rely on GNULIB to try
> > to sanitize the platform inconsistencies where we use POSIX, and assume that
> > any 3rd party libraries we use have done likewise.
> > 
> > Even then, we've tried to avoid using the platform APIs because their designs
> > are often too unsafe to risk using directly (strcat, malloc, free), or are not
> > thread safe (APIs lacking _r variants). So we build our own custom C platform
> > library on top of the base POSIX system, re-inventing the same wheel that every
> > other project written in C invents.
> 
> Why has there never been a truly satisfying standard library for C for
> this kind of stuff? If such a project would exist, this wheel
> re-inventing would be prevented while providing a higher-quality code
> for platform library code.

There are countless "standard" libraries all with their own quirks. Many
of them don't even really address the core safety problems. eg we could
consider glib2 for libvirt, but it just replicates the same awful unsafe
malloc API style that the stdc lib has, with marginal improvement. The
same is true of most other standard libraries. There's essentially no
chance of C ever getting a widely adopted "standard" library at this
point. The only place there's any standardization work is at the POSIX
level and that's waaaaay to low level.

> > Every time we have to do work at the core C platform level, it is
> > diverting time away from doing working managing higher level concepts.
> 
> How often is this the case? I assume that platform code does not change
> that often and will converge into a stable fix-point.

Only if we stop delivering new features.  In pretty much every monthly
release cycle we break platform portability at least once. We have finally
have CI coverage to help us detect with this happens, but you still have
todo the work to make our code portable.

We've got 6000 lines of m4 code for configure.ac in libvirt to cope with
this stuff. If you add in gnulib's m4 code that's 45,000 lines of m4.
If there's one thing C programmers all agree on it is that autoconf is
horrific to write and maintain.  This in turn feeds into 1000's of
#ifdef statements littered throughout the code, which leads to ongoing
maintainence work.

Pretty all this work simply doesn't exist in any modern language which
provides an inherantly portable core runtime.

> > The Solution(s)
> > ===============
> > 
> > [...]
> > 
> > The obvious question / difficulty is deciding how to adopt usage of a new
> > language, without throwing everything away and starting from scratch. It needs
> > to be possible for contributors to continue working on every other aspect of the
> > project while adoption takes place over the long term. Blocking ongoing feature
> > work for prolonged periods of time is not acceptable.
> 
> Yes, I fully concur. But still, I have seen many projects that
> underestimated the amount of work even a partial rewrite in another
> language takes. And in the end, feature development and even bug fixing
> WILL suffer from this transition.
> 
> Maybe it is a good idea to look at the GCC project and their transition
> from C to C++ and learn from their experience beforehand.

Going from C to C++ is a waste of time IMHO. It just gives you a different
and often more complicated ways to shoot yourself in the foot, solving very
few of the core problems with C. You're still using an memory unsafe
language, you still have tonnes of work to write portable code.

> Earlier you talked about the contributor pool. But wouldn't your
> proposal limit this pool even further by actually requiring the
> intersection of the pool of C developers AND Go developers?

Depending on the choice of language this is certainly a valid
concern. It is the primary reason why I would not suggest use
of Rust, because I think that really does limit the pool. With
Go though, any C developers who has written non-trivial C programs
easily has the skills to learn Go code in a matter of days. It would
not be idiomatic Go code, but it doesn't take much more experiance
to learn the Go mindset and start writing high quality code. Go has
been tailored specifically to make the on-ramp as easy as possible.
Mostly you just spend time googling the API docs to learn about the
useful stadnard library APIs you can leverage.

So I don't thnk you limit the contributor pool to the intersection
of developers of the two languages. You really do broaden the pool
to more like the union of the two sets.

> What I would like to see before any rewrite is taken into consideration,
> is an effort to reduce complexity, even on the architectural level. Your
> proposal to split libvirt into set of daemons with specific tasks can
> help here tremendously. In my opinion, a rewrite in another language
> should be a last resort thing if every other options have been
> exhausted, because, from experience, it WILL set a project back.

As mentioned, it would be absolutely critical to ensure that ongoing feature
work can still take place during any transition. That is why I am not
suggesting it as a "big bang" moment. It would be an incremental job over
as long as 5 years or more. There will still be some periods of short term
pain, but over the long term the improved productivity would easily make
up for that. The recent major rewrite of firefox is a great example of
this - they continued releasing on a regular schedule with significant
new features, while in parallel rewriting a large part of their code in
Rust. The end result is a tremendous step forward for the project..

Regards,
Daniel
-- 
|: https://berrange.com      -o-    https://www.flickr.com/photos/dberrange :|
|: https://libvirt.org         -o-            https://fstop138.berrange.com :|
|: https://entangle-photo.org    -o-    https://www.instagram.com/dberrange :|