<div dir="ltr"><div dir="ltr"><div dir="ltr"><br><br><div class="gmail_quote"><div dir="ltr">On Wed, Sep 26, 2018 at 4:25 PM Richard W.M. Jones <<a href="mailto:rjones@redhat.com">rjones@redhat.com</a>> wrote:<br></div><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex">On Wed, Sep 26, 2018 at 02:40:54PM +0200, Fabien Dupont wrote:<br>
> [Adding Tomas Golembiovsky]<br>
><br>
> Well, that's mainly IMS related challenges. We're working on<br>
> OpenStack output support and migration throttling and this implies<br>
> changes to virt-v2v-wrapper. This is then the opportunity to think<br>
> about virt-v2v-wrapper maintenance and feature set. It has been<br>
> created in the first place to simplify interaction with virt-v2v<br>
> from ManageIQ.<br>
<br>
Stepping back here, the upstream community on this mailing list have<br>
no idea what you mean by the terms "IMS", "virt-v2v-wrapper" and<br>
"ManageIQ". I'll try to explain briefly:<br>
<br>
* ManageIQ (<a href="http://manageiq.org/" rel="noreferrer" target="_blank">http://manageiq.org/</a>) = a kind of scriptable, universal<br>
management tool for cloudy things, using Ansible.<br>
<br>
* Ansible = automates remote management of machines using ssh.<br>
<br>
* IMS = an internal Red Hat project to add virt-v2v support to<br>
ManageIQ.<br>
<br>
* virt-v2v-wrapper = a wrapper around virt-v2v which allows it to be<br>
called from Ansible. The reason we need this is because Ansible<br>
doesn't support managing long-running processes like virt-v2v, so we<br>
need to have another component which provides an API which can be<br>
queried remotely, while tending to the long-running virt-v2v behind<br>
the scenes. [Tomáš: Got a link to the code? I can't find it right now]<br></blockquote><div><br></div><div>Thanks for adding explanation. FWIW, Ansible is not involved in IMS.</div><div>There are some rogue code out there</div><div>(<a href="https://github.com/fdupont-redhat/ims-v2v-engine_ansible">https://github.com/fdupont-redhat/ims-v2v-engine_ansible</a>)</div><div>doing things with Ansible, because I like experimenting.</div><div><br></div><div>Virt-v2v-wrapper code is available here:</div><div><a href="https://github.com/oVirt/ovirt-ansible-v2v-conversion-host/blob/master/files/virt-v2v-wrapper.py">https://github.com/oVirt/ovirt-ansible-v2v-conversion-host/blob/master/files/virt-v2v-wrapper.py</a></div><div> </div><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex">
> The first challenge we faced is the interaction with virt-v2v. It's<br>
> highly versatile and proposes a lot of options for input and<br>
> output. The downside of it is that over time is becomes more and<br>
> more difficult to know them all.<br>
<br>
The options are all documented in the manual. I have thought for a<br>
while that we need to tackle the virt-v2v manual: It's too big, and<br>
unapproachable. Really I think we need to split it into topic<br>
sections and rewrite parts of it. Unfortunately I've not had time to<br>
do that so far.<br>
<br>
> And all the error messages are made for human beings, not machines,<br>
> so providing feedback through a launcher, such as virt-v2v-wrapper,<br>
> is difficult.<br>
<br>
This is indeed an issue. Pino recently added enhanced support for the<br>
‘--machine-readable’ option which should address some problems:<br>
<br>
<a href="https://github.com/libguestfs/libguestfs/commit/afa8111b751ed33e1989e6d9bb03928cefa17917" rel="noreferrer" target="_blank">https://github.com/libguestfs/libguestfs/commit/afa8111b751ed33e1989e6d9bb03928cefa17917</a><br>
<br>
If this change still doesn't fully address the issues with automating<br>
virt-v2v then please let us know what specifically can be improved<br>
here.<br></blockquote><div><br></div><div>Thanks for the link. virt-v2v-wrapper is using the standalone call with</div><div>--machine-readable to get the capabilities of virt-v2v. It's used to</div><div>check for --mac option support.</div><div><br></div><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex">
[...]<br>
> For progress, the only way to know what happens is to run virt-v2v<br>
> in debug mode (-v -x) and parse the (very extensive)<br>
> output. Virt-v2v-wrapper does it for us in IMS, but it is merely a<br>
> workaround.<br>
<br>
Right, this is indeed another problem which we should address. I<br>
thought we had an RFE filed for this, but I cannot find it. At the<br>
moment the workaround you mention is very ugly and clunky, but AFAIK<br>
it does work.<br></blockquote><div><br></div><div>You're right. I works. We'll test if the --machine-readable enhances the</div><div>machine experience, and make RFEs if needed.</div><div> </div><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex">
> I'd expect a conversion tool to provide a comprehensive progress,<br>
> such as "I'm converting VM 'my_vm' and more specifically disk X/Y<br>
> (XX%). Total conversion progress is XX%". Of course, I'd also expect<br>
> a machine readable output (JSON, CSV, YAML…). Debug mode ensures we<br>
> have all the data in case of failure, so I don't say remove it, but<br>
> simply add specialized outputs.<br>
<br>
We can discuss debug output vs progress output and formats to use<br>
separately when fixing the above, but yes, point taken.<br>
<br>
> The third challenge was to clean up in case of virt-v2v failure. For<br>
> example, when it fails converting a disk to RHV, it doesn't clean<br>
> the finished and unfinished disks.<br>
<br>
This is a bug (<a href="https://bugzilla.redhat.com/show_bug.cgi?id=1616226" rel="noreferrer" target="_blank">https://bugzilla.redhat.com/show_bug.cgi?id=1616226</a>).<br>
It's been on my to-do list for quite a while , but I haven't got to<br>
it, so patches welcome ...<br></blockquote><div><br></div><div>Thanks for the BZ reference. IIUC, this will clean the disk being converted</div><div>at the kill interruption time. What about the already converted disks for</div><div>a multi-disks VM ? IMO, they should also be removed.</div><div> </div><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex">
> Virt-v2v-wrapper was initially written by RHV team (Tomas) for RHV<br>
> migrations, so it sounded fair(ish). But, extending the outputs to<br>
> OpenStack, we'll have to deal with leftovers in OpenStack too. Maybe<br>
> a cleanup on failure option would be a good idea, with a default to<br>
> false to not break existing behaviour.<br>
<br>
The issue of cleaning up disks in general is a hard one to solve.<br>
<br>
With the OpenStack backend we try our best as long as virt-v2v<br>
exits on a normal failure path:<br>
<br>
<a href="https://github.com/libguestfs/libguestfs/blob/e2bafffce24cd8c0436bf887ee166a3ae2257bbb/v2v/output_openstack.ml#L370-L384" rel="noreferrer" target="_blank">https://github.com/libguestfs/libguestfs/blob/e2bafffce24cd8c0436bf887ee166a3ae2257bbb/v2v/output_openstack.ml#L370-L384</a><br>
<br>
However there are always going to be cases where that is not possible<br>
(eg. virt-v2v segfaults or is kill -9'd or whatever), and in that case<br>
I envisaged for OpenStack some sort of external garbage collector. To<br>
this end, disks which have not been finalized are given a special<br>
description so it should be possible to find them after a full<br>
migration has completed:<br>
<br>
<a href="https://github.com/libguestfs/libguestfs/blob/e2bafffce24cd8c0436bf887ee166a3ae2257bbb/v2v/output_openstack.ml#L386-L392" rel="noreferrer" target="_blank">https://github.com/libguestfs/libguestfs/blob/e2bafffce24cd8c0436bf887ee166a3ae2257bbb/v2v/output_openstack.ml#L386-L392</a><br>
<br>
IIRC virt-v2v-wrapper is sending kill -9 to virt-v2v, which it should<br>
not do.<br></blockquote><div><br></div><div>It's not virt-v2v-wrapper that kills virt-v2v, it's ManageIQ. We have the</div><div>PID from virt-v2v-wrapper state file. What would be the preferred way</div><div>to interrupt it ?</div><div> </div><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex">
> The fourth challenge is to limit the resources allocated to virt-v2v<br>
> during conversion, because concurrent conversions may have a huge<br>
> impact on conversion host performance. In the case of an oVirt host,<br>
> this can impact the virtual machines that run on it. This is not<br>
> covered yet by the wrapper, but implementation will likely be based<br>
> on Linux cgroups and tc.<br>
<br>
Right, sounds sensible.<br>
<br>
> The wrapper also adds an interesting feature: both virt-v2v and<br>
> virt-v2v-wrapper run daemonized and we can asynchronously poll the<br>
> progress. This is really key for IMS (and maybe for others): this<br>
> allows us to start as many conversions in parallel as needed and<br>
> monitor them. Currently, the Python code forks and detaches itself,<br>
> after providing the paths to the state file. In the discussion about<br>
> cgroups, it was mentioned that systemd units could be used, and it<br>
> echoes with the daemonization, as systemd-run allows running<br>
> processes under systemd and in their own slice, on which cgroups<br>
> limits can be set.<br>
<br>
"the Python code" meaning the code in virt-v2v-wrapper?<br></blockquote><div><br></div><div>Yes, it is.</div><div><br></div><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex">I also agree that using a systemd temporary unit is the way to go<br>
here. As well as providing a natural way to limit the resources used<br>
by virt-v2v (since systemd unit implies a cgroup), it also solves the<br>
problems around logging and collecting debug logs.<br>
<br>
> About the evolution of virt-v2v-wrapper that I'm going to describe,<br>
> let me state that this is my personal view and it endorses only<br>
> myself.<br>
><br>
> I would like to see the machine-to-machine interaction, logging and<br>
> cleanup in virt-v2v itself because it is valuable to everyone, not<br>
> only IMS.<br>
<br>
Here's I think where we are going to disagree. virt-v2v is a command<br>
line tool, with lots of users outside of the internal IMS Red Hat<br>
project. Here in the upstream community I'm afraid we make decisions<br>
which are best for all our users, not for one particular user!<br></blockquote><div><br></div><div>I agree with you. That's why I propose that M2M interaction (e.g.</div><div>--machine-readable), logging (format and output options, not only</div><div>journald capture) and cleanup (handling interruption) are done at</div><div>the virt-v2v level. IMO, these would be beneficial to everybody.</div><div><br></div><div>As you say, invocation via systemd is useful for IMS, and</div><div>documenting it would make it valuable to others. The other three</div><div>topics are not linked to systemd and should be addressed on their</div><div>own.</div><div> </div><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex">
But in any case I don't see how adding systemd unit support to<br>
virt-v2v itself helps very much. It's really easy to run virt-v2v in<br>
a systemd unit -- see the attached email for full details.<br>
<br>
This gains all the benefits I mention above and is hardly any effort<br>
at all. You can even adjust the properties on the fly.<br>
<br>
(I will admit this is really obscure and undocumented, it took me<br>
quite a lot of time last month to work it out. We should add this to<br>
the virt-v2v documentation, but at least the docs are available on the<br>
mailing list now.)<br>
<br>
> I would also like to convert virt-v2v-wrapper to a conversion API<br>
> and Scheduler service. The idea is that it would provide an<br>
> as-a-Service endpoint for conversions, that would allow creation of<br>
> conversion jobs (POST), fetching of the status (GET), cancelation of<br>
> a conversion (DELETE) and changing of the limits (PATCH). In the<br>
> background, a basic scheduler would simply ensure that all the jobs<br>
> are running. Each virt-v2v process would be run as a systemd unit<br>
> (journald could capture the debug output), so that it is independent<br>
> from the API and Scheduler processes.<br>
<br>
This sounds like an interesting and useful evolution of the wrapper,<br>
and we should try to add pieces to virt-v2v to make it easier to run<br>
under the wrapper, but at the end of the day virt-v2v is a command<br>
line tool used by many different projects and purposes so actually<br>
adding all this to virt-v2v itself is a non-starter.<br></blockquote><div><br></div><div>Again, agreed. That's why I talk about an evolution of</div><div>virt-v2v-wrapper. Some of the clunky workarounds virt-v2v-wrapper</div><div>implements should be handled by virt-v2v. The launch and</div><div>monitoring of virt-v2v is out of scope of the virt-v2v command.</div><div> </div><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex">> I know that I can propose patches for changes to virt-v2v, or at<br>
> least file RFEs in Bugzilla (my developer skills and programing<br>
> languages breadth are limited). For the evolved wrapper, my main<br>
> concern is its housing and maintenance. It doesn't work only for<br>
> oVirt, so having its lifecycle tied to oVirt doesn't seem relevant<br>
> in the long term. In fact, it can be for any virt-v2v output, so my<br>
> personal opinion is that it should live in the virt-v2v ecosystem<br>
> and follow it's lifecycle. As for its maintenance, we still have to<br>
> figure out who will be responsible for it, i.e. who will be able to<br>
> dedicate time to it.<br>
<br>
There's certainly a case for making the wrapper into a standalone<br>
project, with a proper upstream etc. It could even be shipped under<br>
the libguestfs umbrella. But that's Tomáš's domain so I leave it up<br>
to him to decide what to do.<br></blockquote><div><br></div><div>That's why I added him to this thread :) </div><div> </div><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex">
Rich.<br>
<br>
-- <br>
Richard Jones, Virtualization Group, Red Hat <a href="http://people.redhat.com/~rjones" rel="noreferrer" target="_blank">http://people.redhat.com/~rjones</a><br>
Read my programming and virtualization blog: <a href="http://rwmj.wordpress.com" rel="noreferrer" target="_blank">http://rwmj.wordpress.com</a><br>
Fedora Windows cross-compiler. Compile Windows programs, test, and<br>
build Windows installers. Over 100 libraries supported.<br>
<a href="http://fedoraproject.org/wiki/MinGW" rel="noreferrer" target="_blank">http://fedoraproject.org/wiki/MinGW</a><br>
<br><br><br>---------- Forwarded message ----------<br>From: "Richard W.M. Jones" <<a href="mailto:rjones@redhat.com" target="_blank">rjones@redhat.com</a>><br>To: <a href="mailto:v2v-devel@redhat.com" target="_blank">v2v-devel@redhat.com</a><br>Cc: <br>Bcc: <br>Date: Wed, 22 Aug 2018 14:42:51 +0100<br>Subject: Limiting virt-v2v block I/O using cgroups - network I/O still unknown<br>Turns out this is fairly easy, although quite obscure.<br>
<br>
Just use ‘systemd-run --pipe’ to run the virt-v2v command in a cgroup.<br>
The ‘--pipe’ option ensures it is still connected to stdin/stdout/<br>
stderr (but see below).<br>
<br>
$ systemd-run --user --pipe \<br>
-p BlockIOWriteBandwidth="/dev/sda2 1K" \<br>
virt-v2v -i disk /var/tmp/fedora-27.img -o local -os /var/tmp<br>
<br>
Running as unit: run-u4429.service<br>
[ 0.0] Opening the source -i disk /var/tmp/fedora-27.img<br>
[ 0.0] Creating an overlay to protect the source from being modified<br>
etc.<br>
<br>
See systemd.resource-control(5) for a list of controls. If you are<br>
experimenting with this then it is easier to start with a command like<br>
‘sleep 1000000’ rather than using virt-v2v.<br>
<br>
Some notes:<br>
<br>
(1) systemd-run changes the directory to ‘/’ so all path parameters to<br>
virt-v2v must be absolute.<br>
<br>
(2) If using something called "cgroup v2" you have to use<br>
IOWriteBandwithMax. However even though I'm using a very recent<br>
Fedora & Linux kernel, I'm apparently not using cgroup v2.<br>
<br>
(3) You can modify the settings on the fly using:<br>
<br>
$ systemctl [--user] set-property --runtime run-uXXX.service \<br>
BlockIOWriteBandwith="/dev/sda2 NEW_SETTING"<br>
<br>
where run-uXXX.service is the service name printed by systemd-run<br>
before it starts virt-v2v.<br>
<br>
Also:<br>
<br>
$ systemctl --user show run-u4466.service | grep BlockIO<br>
BlockIOAccounting=no<br>
BlockIOWeight=[not set]<br>
StartupBlockIOWeight=[not set]<br>
BlockIOWriteBandwidth=/dev/sda2 10000<br>
<br>
Note you have to enable BlockIOAccounting to collect stats.<br>
<br>
Also:<br>
<br>
systemctl [--user] status run-uXXX.service<br>
<br>
to read information about the status of the service.<br>
<br>
Also there are other systemd/cgroup tools like ‘systemd-cgtop’ and<br>
‘systemd-cgls’ which may be useful to administrators.<br>
<br>
(4) Specifying the block device is a PITA. My reading of the<br>
documentation makes me think you can use filesystem paths instead of<br>
block device names, but it didn't work for me<br>
(<a href="https://github.com/systemd/systemd/issues/9908" rel="noreferrer" target="_blank">https://github.com/systemd/systemd/issues/9908</a>).<br>
<br>
- * - * - * -<br>
<br>
Network prioritization (what we actually care about) is quite a lot<br>
more complex. First of all, documentation everywhere refers to<br>
net_cls (NetClass), but that is apparently obsolete. The new thing is<br>
net_prio, but virtual nothing talks about how to use that. In any<br>
case we'd use it in conjunction with ‘tc’.<br>
<br>
- * - * - * -<br>
<br>
While looking into this I wondered if it wouldn't be better to run<br>
virt-v2v in a proper systemd unit. We'd use systemd to set up<br>
logging. The wrapper would simply become a small script that controls<br>
the unit through systemctl.<br>
<br>
Rich.<br>
<br>
-- <br>
Richard Jones, Virtualization Group, Red Hat <a href="http://people.redhat.com/~rjones" rel="noreferrer" target="_blank">http://people.redhat.com/~rjones</a><br>
Read my programming and virtualization blog: <a href="http://rwmj.wordpress.com" rel="noreferrer" target="_blank">http://rwmj.wordpress.com</a><br>
virt-top is 'top' for virtual machines. Tiny program with many<br>
powerful monitoring features, net stats, disk stats, logging, etc.<br>
<a href="http://people.redhat.com/~rjones/virt-top" rel="noreferrer" target="_blank">http://people.redhat.com/~rjones/virt-top</a><br>
</blockquote></div><br clear="all"><div><br></div>-- <br><div dir="ltr" class="gmail_signature"><div dir="ltr"><div><div dir="ltr"><div dir="ltr"><div dir="ltr"><div dir="ltr"><div dir="ltr"><div dir="ltr"><div dir="ltr"><div dir="ltr"><div dir="ltr"><div dir="ltr"><div dir="ltr"><p style="margin:0px;padding:0px"><b style="font-family:arial,helvetica,sans-serif;font-size:small"><font color="#cc0000"><span style="margin:0px;padding:0px">Fabien</span> <span style="margin:0px;padding:0px">Dupont</span></font></b><br></p><p style="color:rgb(0,0,0);margin:0px;padding:0px"></p><p style="font-size:small;color:rgb(0,0,0);margin:0px;padding:0px"><span style="font-family:arial,helvetica,sans-serif;font-size:x-small">PRINCIPAL SOFTWARE ENGINEER</span><font size="1" face="arial, helvetica, sans-serif"><br style="margin:0px;padding:0px"></font></p><p style="font-size:small;margin:0px;padding:0px"><font face="arial, helvetica, sans-serif" size="1" color="#000000">Red Hat - Solutions Engineering</font></p><p style="margin:0px;padding:0px"><font size="1" face="arial, helvetica, sans-serif"><span style="margin:0px;padding:0px"><span style="margin:0px;padding:0px"><a href="mailto:fabien@redhat.com" target="_blank"><font color="#0b5394">fabien@redhat.com</font></a></span><font color="#000000"> </font></span><span style="margin:0px;padding:0px"><font color="#000000">M: </font><a href="tel:+33662784971" style="margin:0px;padding:0px" target="_blank"><font color="#0b5394">+33 (0) 662 784 971</font></a></span></font></p><p style="margin:0px;padding:0px"><span style="margin:0px;padding:0px"><font size="1" face="arial, helvetica, sans-serif"><a href="http://redhat.com" style="color:rgb(0,0,0)" target="_blank"><img src="https://www.redhat.com/profiles/rh/themes/redhatdotcom/img/logo-red-hat-black.png" width="96" height="30"></a><font color="#000000"> </font><span style="margin:0px;padding:0px"><font color="#cc0000"><b>TRIED. TESTED. TRUSTED.</b></font></span></font></span></p><p style="color:rgb(0,0,0);margin:0px;padding:0px"><font size="1" face="arial, helvetica, sans-serif"><span style="margin:0px;padding:0px"></span></font></p><div style="color:rgb(0,0,0);margin:0px;padding:0px"><div style="margin:0px;padding:0px"><font size="1" face="arial, helvetica, sans-serif">Twitter: <a href="https://twitter.com/redhatway" target="_blank">@redhatway</a> | Instagram: <a href="https://www.instagram.com/redhatinc/" target="_blank">@redhatinc</a> | Snapchat: @redhatsnaps</font></div></div></div></div></div></div></div></div></div></div></div></div></div></div></div></div></div></div></div>