[libvirt] Redesigning Libvirt: Adopting use of a safe language

Sun Nov 19 23:24:22 UTC 2017

On Tue, Nov 14, 2017 at 05:27:01PM +0000, Daniel P. Berrange wrote:

[...]

>I don't have direct experiance in Rust, but it has the same kind of benefits over
>C as Go does, again without the downsides of languages like Python or Java. There
>are some interesting unique features to Rust that can be important to some apps.
>In particular it does not use garbage collection, instead the user must still do
>manual memory management as you would with C/C++. This allows Rust to be used in
>performance critical cases where it is unacceptable to have a garbage collector
>run. Despite a requirement for manual allocation/deallocation, Rust still
>provides a safe memory model. This approach of avoiding abstractions which will
>introduce performance overhead is a theme of Rust. The cost of such an approach
>is that development has a higher learning curve and ongoing cost in Rust, as
>compared to Go.
>
>I don't believe that the unique features of Rust, over Go, are important to the
>needs of libvirt. eg while for QEMU it would be critical to not have a GC
>doing asynchronous memory deallocation, this is not at all important to libvirt.
>In fact precisely the opposite, libvirt would benefit much more from having GC
>take care of deallocation, letting developers focus attention other areas. In
>general, as from having a memory safe language, what libvirt would most benefit
>from is productivity gains & ease of contribution. This is the core competancy
>of Go, and why it is the right choice for usage in libvirt.
>

Even though I agree with you on most of the opinions of this thread, I
must engage in a discussion on this part.  I started working on libvirt
because I liked the fact that it is in C and needs to work with the
low-level concepts and interfaces.  I like python for rapid prototyping,
but I used to be against new, "hype" languages that were popping up here
and there.  I must say, on the other hand, that I was surprised how
choice of language can influence the contributor flow (mostly incoming).
However I heard about this mostly happening in the ruby and node.js
communities.  Along the way, I quickly stopped thinking about Rust and
Go as hype languages.  Well, so much for my background...

I'm going to approach the elephant in the room, but I'm not the one to
defend the opinion as my understanding, as well as experience, is
limited.  On the other hand, that could show you that it is not that
hard for a newbie to get the hang of it.  I'm saying this because I
might get some things wrong, but I expect the wrong information to be
related mainly to Go.

So the first thing I disagreed on is that in Rust you do manual
allocations.  In fact, you don't.  Or, depending on the point of view,
you do less or the same amount of manual allocation than in Go.  What is
the clear win for Rust it the concept of ownership and it's related to
the allocation mentioned before.

I am standing strongly behind the opinion that the learning curve of
Rust is definitely worth it.  And coming from the C world, it is easy to
understand.  To me, it is very easy to explain that concept to great
detail to someone who has background in libvirt.  And the big benefit
(and still a huge opportunity for improvement WRT optimizations) is that
the compiler must know about it and so it is resolved compile-time.
Dereferencing or destructors are run at the end of their scope,
automatically.  You can nicely see that when realizing that Rust doesn't
need any `defer` as Go has.

Sure, Rust doesn't have green threads implementation, however there
should be support for that in some library.  I used vague wording due to
the fact that I haven't yet found it, however it is mentioned in
official docs.  Apparently it was removed from the standard library due
to requirements on runtime size.

Rust's slogan (or one of them?) is "fearless concurrency".  It builds
upon the ideas from Go, however, having more information it can work
with it better.  It could help us considering how many problems we have
with reference counting and similar.

So much for defending Rust.  Long story short, I think we could benefit
a tiny (well, IMHO not really tiny) bit more if we go that route
instead.  Now for some more opinions I have.  Stay with me.

Not considering the Linux kernel, which has their own address sanitizer,
randomizer and god-knows-whatifier, there is still plethora or projects
that are successful in C and I don't see how they would need to adapt or
die.  I know we have bunch of stuff that needs to be fixed and the
current approach doesn't make it easy.  I, myself, thought about
proposing new version of libvirt, which would be a rewrite of the
current one.  Thanks to that I have some reasons why I didn't do that.

One of the things is that the kinks we have can be ironed out in C as
well.  It might be easier in other languages, but it is harder when you
have to switch to one.  We have bunch of code dealing with backwards
compatibility.  And I argue that this is something that causes issues on
its own.  What's even worse, IMHO, is that we are so much feature-driven
that there is no time for any ironing.  I see too much potential for
refactoring in various parts of libvirt that will never see the lights
of day because we need X to be implemented.  And contributors sending
feature requests that they fail to maintain later don't help much with
that.  Maybe we could fix this by saying the next Y releases will just
be bugfix releases.  Maybe we could help bringing new contributors by
devoting some of our time to do an actual change that will make them
want to help us more.  I know some of you will be sick and tired hearing
about Rust once more, but have you heard about how much their community
is inclusion-oriented?  I guess what I'm trying to say is that there are
other (and maybe less disruptive) ways to handle the current problems we
are facing.

And then there are the "issues" with Go (and unfortunately some with
Rust as well :'( ).

Lot of the code for libraries is written with permissive licences, but
if there is some that is LGPL-incompatible we can't use them.  And in
ecosystems such as Rust and Go there are fewer alternatives, so we might
not find one that we'll be able to use.  If that happens, there goes
bunch of our time.  Like nothing.

How do we deal with problems/bugs in dependence libraries?  I know, all
the projects are pretty new, so they might be nicer to contributors.  If
they are not, we might need to fork or rewrite the code.  Bam, another
chance of losing workforce.

If I may, I will ask some things about Go that I'm not that familiar
with and I think they are pretty important.

How does Go handle updates in dependency libs?  Does it automatically
pull newest version from public repositories where some unknown person
can push whatever they want?  Or can they be hash- or version-bound?

The build process is that all the binaries are static, right?  Or all
the go code is static and it only has dynamic dependencies on C
libraries? In such a project as libvirt, wouldn't that mean that the
processes we run will be pretty heavy-weight?

How is it with rebuilding after a small change.  I know Go is good when
it comes to compilation times.  That might be something that people
might like a lot.  Especially those who are trying to shave off every
second of compilation.  However if you cannot use ccache and you always
need to rebuild everything, it might increase the build-time quite a
lot, even thought indirectly.

Since there were some hateful opinions about Go, let me add my share as
well.  Just so we are on the same page and I don't miss saying anything.
Not that these opinions would be as important as the ones above.

You can't have /tmp mounted with noexec option unless you have TMPDIR
set to some other directory.  And sometimes not even in that case.  I
guess non-issue for some distros, but bunch of people deal with that and
if seems like something that could be taken care of in Go itself and it
is just not.

You can't just clone a repo, cd into it and build it.  You have to get
the dependencies in a special manner, for that you have to have GOPATH
set and based on that you have to have your directories setup similarly
to what Go expects, then you need GOBIN if you want to build something
and other stuff that's just not nice and doesn't make much sense.  At
least for newcomers.  Simply the fact that it seems to me like Go is
trying to go against the philosophy of "Do one thing and do it well".  I
know everyone is about having the build described in the same language
as the project, but what comes out of it is not something I prefer.

I'm not against using another language to make some stuff better.  I
guess it is kind of visible from the mail that I like Rust, but I'm not
against other languages as well.  I just want this to be full on
discussion and I want my opinion to be expressed.  Thanks for
_listening_ ;)

Have a nice day,
Martin
-------------- next part --------------
A non-text attachment was scrubbed...
Name: signature.asc
Type: application/pgp-signature
Size: 833 bytes
Desc: Digital signature
URL: <http://listman.redhat.com/archives/libvir-list/attachments/20171120/a5b20e8d/attachment-0001.sig>