[libvirt] Redesigning Libvirt: Adopting use of a safe language

Thu Nov 16 16:56:52 UTC 2017

On Thu, Nov 16, 2017 at 02:58:33PM +0100, Peter Krempa wrote:
> On Tue, Nov 14, 2017 at 17:27:01 +0000, Daniel Berrange wrote:
> > The Problem(s)
> > ==============
> 
> Note at first: This is a personal opinion. I'm not discrediting any
> advantages a different language might have in technical sense.
> 
> I'm against this. Go is a utterly ugly language. It looks as the authors
> wanted to be so hip so that they had to do a lot of stuff backwards.
> Literally. Just look at variable declaration. While some concepts might
> be cool, the rest of the stuff makes my eyes bleed. I hate it. Seriously
> who on earth thought that the network socket connection API should be
> called 'dial'.

I have a hard time calling Go ugly when the comparison is C. The things
we do with C and macros are horrific by comparison. Go has very strong
readability. Variable declartions syntax has little to no impact on
ability to use the language productive manner. Likewise API nameing is
just a matter of taste. Sure 'dial' is odd, but there's 1000's of
wierdly named methods across APIs used in any language. 

> Any language allows for people to do mistakes. You can eliminate the
> very common mistakes, but they are comparatively easy to spot and fix.
> You can't eliminate the very subtle logic problems, which usually linger
> a long time in libvirt.

I think evidence of libvirt and every other non-trivial C program sshows
that when it comes to memory management mistakes it is essentially to
eliminate the common mistakes. People make the same mistake over & over
again, even experts in the language. We have tools that help us spot some
of them, but nowhere near all of them, as evidenced by the constant stream
of crashes we have to fix in libvirt which coverity doesn't tell us about.

I agree that subtle logic problems can exist in any language and libvirt
has its fair share of those and they won't be magically solved by a
different language. We waste a hell of alot of resources on dealing with
problems inherant in  the usage of C (whether platform portability, or
mmemory management or reinventing an object system). By not wasting that
time, we will ultimately be able to put more time into identifying and
fixing the subtle logic problems.

> While the language itself may get rid of one layer of compatibility
> functions/libraries like gnulib, most of our uses are hidden in utility
> functions. Getting rid of it will remove some stuff but we will end up
> with a different pile of utility code in a different language. It may be
> simpler in the end but it will be there and will be used similarly as we
> use the current utility code. There's no clear win in my opinion here.

I believe it will be a major net win. We have many 10's of thousands
of lines of autoconf, and automake code, and code littered if #ifdefs
many combinations of which are never tested properly. We have 65,000
lines of code just for parsing & formatting XML. The vast majority of
that can be automatically handled in Go with mere annotations on
struct fields, and the result is far safer because it would get XML
entity escaping right without any effort.  ON networking stuff we've
written alot of code to help us integrate with sockets, TLS, SSH
and so on, because while there are libraries for this none of the
them provide the same API. Again the need to write that kind of
boilerplating all goes away because there's a proper extensible
I/O framework that. The amount of time we spend writing infrastructure
in C that already exists in other language's standard libraries is
very significant. 

> Then there's the rewrite. While the language may allow calling of C code
> and being called from C code we will need to draw a line somewhere.
> Adding new Go code into existing C code will be ugly. Rejecting someones
> contribution because it's in C is wrong. Also just spending time writing
> the same stuff in a supposedly cooler language feels wrong to me. I can
> see the value in adding new stuff, but it's a waste to rewrite existing
> stuff.

I'm not suggesting to unconditionally reject people's contributions
in C. If the area of code they want to contribute to is still using
C, then they should certainly write submit patches in the same way.
This is a key reasn why I say we shouldn't attempt any "stop the
world & rewrite" big bang - it if far too distruptive and prevents
people doing other useful work in parallel. A conversion would have
to be incrementally evolved over a prolonged period of time, so we
can still satisfy ongoing feature requests & bug fixes in parallel.

This is not about using a cooler language, it is about using a
language which will let us provide a much more reliable libvirt
which doesn't randomly crash & leak memory, and where we can focus
our dev resources on solving the interesting problems that libvirt
has, instead of wasting time dealing with the problems of C.

> Then there's debugging. Official docs hint to use GDB but in the same
> paragraph note that GDB does not work entirely well with Go programs.
> [1] There might be other tools like Delve [2] which are still under
> heavy development. I'm not entirely persuaded in this area.

That is certainly a fair point. The flipside is that a great many
of the reasons for needing GDB / valgrind / coverity / etc in C
are removed by virtue of having a safe memory management design.
In the intermediate where we have a mix of C and Go in the same
process though, I accept it would make some debugging tasks harder.
I don't think that's a show stopping problem, because over the
long term it'd be a net win.

> I don't like this. Don't expect me to write any Go code.

I don't know if I'll change your mind, but I really encourage people to
actually look beyond the obvious surface differences. As languages go
it is explicitly designed to be easy to understand & familiar for C
developers, much more so than any other competing languages out there. 
By having a truly compiled language, as opposed to a VM like Java, or a
interpretor like python/perl/ruby/etc, you're still close to the
machine layer as you are in C. You can still easily call into C APIs
where you need to with ease as the basic scalar data types are all very
similar to C, while giving you much easier to use compound data types
(lists, strings, maps, etc). Being free from the maddness of manual
memory allocation/deallocation/overruns greatly improves productivity
and reliability of the code, and you no longer waste so much time
reinventing the wheel whether for cross-platform porting or basic
library APIs. I struggle to think of any compelling reason to continue
to use C, aside from the fact that it is what I've always worked with
for ~20 years. Many languages have come up in that time, but none has
been able to credibly replace C for low overhead systems programming
work until Go and Rust came along.

Regards,
Daniel
-- 
|: https://berrange.com      -o-    https://www.flickr.com/photos/dberrange :|
|: https://libvirt.org         -o-            https://fstop138.berrange.com :|
|: https://entangle-photo.org    -o-    https://www.instagram.com/dberrange :|