<div dir="ltr"><div class="gmail_quote"><div dir="ltr">On Thu, Mar 8, 2018 at 11:37 AM Richard W.M. Jones <<a href="mailto:rjones@redhat.com">rjones@redhat.com</a>> wrote:<br></div><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex">PROBLEMS:<br>
- Target cluster defaults to "Default".<br>
- Using Insecure = True, is that bad?<br>
- -of qcow2 does not work, with multiple problems<br>
- Need to attach disks to VMs somehow<br>
<br>
This adds a new output mode to virt-v2v. virt-v2v -o rhv-upload<br>
streams images directly to an oVirt or RHV >= 4 Data Domain using the<br>
oVirt SDK v4. It is more efficient than -o rhv because it does not<br>
need to go via the Export Storage Domain, and is possible for humans<br>
to use unlike -o vdsm.<br>
<br>
The implementation uses the Python SDK (‘ovirtsdk4’ module). An<br>
nbdkit Python 3 plugin translates NBD calls from qemu into HTTPS<br>
requests to oVirt via the SDK.<br>
---<br>
.gitignore | 3 +<br>
v2v/Makefile.am | 26 ++-<br>
v2v/<a href="http://cmdline.ml" rel="noreferrer" target="_blank">cmdline.ml</a> | 38 ++++<br>
v2v/embed.sh | 45 ++++<br>
v2v/<a href="http://output_rhv_upload.ml" rel="noreferrer" target="_blank">output_rhv_upload.ml</a> | 330 ++++++++++++++++++++++++++++++<br>
v2v/output_rhv_upload.mli | 27 +++<br>
v2v/output_rhv_upload_createvm_source.mli | 19 ++<br>
v2v/output_rhv_upload_plugin_source.mli | 19 ++<br>
v2v/output_rhv_upload_precheck_source.mli | 19 ++<br>
v2v/rhv-upload-createvm.py | 85 ++++++++<br>
v2v/rhv-upload-plugin.py | 248 ++++++++++++++++++++++<br>
v2v/rhv-upload-precheck.py | 72 +++++++<br>
v2v/virt-v2v.pod | 105 ++++++++--<br>
13 files changed, 1021 insertions(+), 15 deletions(-)<br>
<br>
diff --git a/.gitignore b/.gitignore<br>
index d72447d1d..211376eef 100644<br>
--- a/.gitignore<br>
+++ b/.gitignore<br>
@@ -654,6 +654,9 @@ Makefile.in<br>
/utils/qemu-speed-test/qemu-speed-test<br>
/v2v/.depend<br>
/v2v/oUnit-*<br>
+/v2v/<a href="http://output_rhv_upload_createvm_source.ml" rel="noreferrer" target="_blank">output_rhv_upload_createvm_source.ml</a><br>
+/v2v/<a href="http://output_rhv_upload_plugin_source.ml" rel="noreferrer" target="_blank">output_rhv_upload_plugin_source.ml</a><br>
+/v2v/<a href="http://output_rhv_upload_precheck_source.ml" rel="noreferrer" target="_blank">output_rhv_upload_precheck_source.ml</a><br>
/v2v/real-*.d/<br>
/v2v/real-*.img<br>
/v2v/real-*.xml<br>
diff --git a/v2v/Makefile.am b/v2v/Makefile.am<br>
index c2eb31097..392eaefcc 100644<br>
--- a/v2v/Makefile.am<br>
+++ b/v2v/Makefile.am<br>
@@ -22,12 +22,19 @@ generator_built = \<br>
uefi.mli<br>
<br>
BUILT_SOURCES = \<br>
- $(generator_built)<br>
+ $(generator_built) \<br>
+ <a href="http://output_rhv_upload_createvm_source.ml" rel="noreferrer" target="_blank">output_rhv_upload_createvm_source.ml</a> \<br>
+ <a href="http://output_rhv_upload_plugin_source.ml" rel="noreferrer" target="_blank">output_rhv_upload_plugin_source.ml</a> \<br>
+ <a href="http://output_rhv_upload_precheck_source.ml" rel="noreferrer" target="_blank">output_rhv_upload_precheck_source.ml</a><br>
<br>
EXTRA_DIST = \<br>
$(SOURCES_MLI) $(SOURCES_ML) $(SOURCES_C) \<br>
<a href="http://copy_to_local.ml" rel="noreferrer" target="_blank">copy_to_local.ml</a> \<br>
copy_to_local.mli \<br>
+ embed-code.sh \<br>
+ rhv-upload-createvm.py \<br>
+ rhv-upload-plugin.py \<br>
+ rhv-upload-precheck.py \<br>
<a href="http://v2v_slow_unit_tests.ml" rel="noreferrer" target="_blank">v2v_slow_unit_tests.ml</a> \<br>
v2v-slow-unit-tests.sh \<br>
<a href="http://v2v_unit_tests.ml" rel="noreferrer" target="_blank">v2v_unit_tests.ml</a> \<br>
@@ -64,6 +71,10 @@ SOURCES_MLI = \<br>
output_null.mli \<br>
output_qemu.mli \<br>
output_rhv.mli \<br>
+ output_rhv_upload.mli \<br>
+ output_rhv_upload_createvm_source.mli \<br>
+ output_rhv_upload_plugin_source.mli \<br>
+ output_rhv_upload_precheck_source.mli \<br>
output_vdsm.mli \<br>
parse_ovf_from_ova.mli \<br>
parse_libvirt_xml.mli \<br>
@@ -116,6 +127,10 @@ SOURCES_ML = \<br>
<a href="http://output_local.ml" rel="noreferrer" target="_blank">output_local.ml</a> \<br>
<a href="http://output_qemu.ml" rel="noreferrer" target="_blank">output_qemu.ml</a> \<br>
<a href="http://output_rhv.ml" rel="noreferrer" target="_blank">output_rhv.ml</a> \<br>
+ <a href="http://output_rhv_upload_createvm_source.ml" rel="noreferrer" target="_blank">output_rhv_upload_createvm_source.ml</a> \<br>
+ <a href="http://output_rhv_upload_plugin_source.ml" rel="noreferrer" target="_blank">output_rhv_upload_plugin_source.ml</a> \<br>
+ <a href="http://output_rhv_upload_precheck_source.ml" rel="noreferrer" target="_blank">output_rhv_upload_precheck_source.ml</a> \<br>
+ <a href="http://output_rhv_upload.ml" rel="noreferrer" target="_blank">output_rhv_upload.ml</a> \<br>
<a href="http://output_vdsm.ml" rel="noreferrer" target="_blank">output_vdsm.ml</a> \<br>
<a href="http://inspect_source.ml" rel="noreferrer" target="_blank">inspect_source.ml</a> \<br>
<a href="http://target_bus_assignment.ml" rel="noreferrer" target="_blank">target_bus_assignment.ml</a> \<br>
@@ -126,6 +141,15 @@ SOURCES_C = \<br>
libvirt_utils-c.c \<br>
qemuopts-c.c<br>
<br>
+# These files are generated and contain rhv-upload-*.py embedded as an<br>
+# OCaml string.<br>
+<a href="http://output_rhv_upload_createvm_source.ml" rel="noreferrer" target="_blank">output_rhv_upload_createvm_source.ml</a>: rhv-upload-createvm.py<br>
+ ./embed.sh code $^ $@<br>
+<a href="http://output_rhv_upload_plugin_source.ml" rel="noreferrer" target="_blank">output_rhv_upload_plugin_source.ml</a>: rhv-upload-plugin.py<br>
+ ./embed.sh code $^ $@<br>
+<a href="http://output_rhv_upload_precheck_source.ml" rel="noreferrer" target="_blank">output_rhv_upload_precheck_source.ml</a>: rhv-upload-precheck.py<br>
+ ./embed.sh code $^ $@<br>
+<br>
if HAVE_OCAML<br>
<br>
bin_PROGRAMS = virt-v2v virt-v2v-copy-to-local<br>
diff --git a/v2v/<a href="http://cmdline.ml" rel="noreferrer" target="_blank">cmdline.ml</a> b/v2v/<a href="http://cmdline.ml" rel="noreferrer" target="_blank">cmdline.ml</a><br>
index d725ae022..c53d1703b 100644<br>
--- a/v2v/<a href="http://cmdline.ml" rel="noreferrer" target="_blank">cmdline.ml</a><br>
+++ b/v2v/<a href="http://cmdline.ml" rel="noreferrer" target="_blank">cmdline.ml</a><br>
@@ -65,6 +65,8 @@ let parse_cmdline () =<br>
let output_password = ref None in<br>
let output_storage = ref None in<br>
let password_file = ref None in<br>
+ let rhv_cafile = ref None in<br>
+ let rhv_direct = ref false in<br>
let vddk_config = ref None in<br>
let vddk_cookie = ref None in<br>
let vddk_libdir = ref None in<br>
@@ -143,6 +145,8 @@ let parse_cmdline () =<br>
| "disk" | "local" -> output_mode := `Local<br>
| "null" -> output_mode := `Null<br>
| "ovirt" | "rhv" | "rhev" -> output_mode := `RHV<br>
+ | "ovirt-upload" | "ovirt_upload" | "rhv-upload" | "rhv_upload" -><br>
+ output_mode := `RHV_Upload<br>
| "qemu" -> output_mode := `QEmu<br>
| "vdsm" -> output_mode := `VDSM<br>
| s -><br>
@@ -229,6 +233,9 @@ let parse_cmdline () =<br>
[ L"print-source" ], Getopt.Set print_source,<br>
s_"Print source and stop";<br>
[ L"qemu-boot" ], Getopt.Set qemu_boot, s_"Boot in qemu (-o qemu only)";<br>
+ [ L"rhv-cafile" ], Getopt.String ("ca.pem", set_string_option_once "--rhv-cafile" rhv_cafile),<br>
+ s_"For -o rhv-upload, set ‘ca.pem’ file";<br>
+ [ L"rhv-direct" ], Getopt.Set rhv_direct, s_"Use direct transfer mode";<br>
[ L"root" ], Getopt.String ("ask|... ", set_root_choice),<br>
s_"How to choose root filesystem";<br>
[ L"vddk-config" ], Getopt.String ("filename", set_string_option_once "--vddk-config" vddk_config),<br>
@@ -322,6 +329,8 @@ read the man page virt-v2v(1).<br>
let password_file = !password_file in<br>
let print_source = !print_source in<br>
let qemu_boot = !qemu_boot in<br>
+ let rhv_cafile = !rhv_cafile in<br>
+ let rhv_direct = !rhv_direct in<br>
let root_choice = !root_choice in<br>
let vddk_options =<br>
{ vddk_config = !vddk_config;<br>
@@ -546,6 +555,35 @@ read the man page virt-v2v(1).<br>
Output_rhv.output_rhv os output_alloc,<br>
output_format, output_alloc<br>
<br>
+ | `RHV_Upload -><br>
+ let output_conn =<br>
+ match output_conn with<br>
+ | None -><br>
+ error (f_"-o rhv-upload: use ‘-oc’ to point to the oVirt or RHV server REST API URL, which is usually <a href="https://servername/ovirt-engine/api" rel="noreferrer" target="_blank">https://servername/ovirt-engine/api</a>")<br>
+ | Some oc -> oc in<br>
+ (* In theory we could make the password optional in future. *)<br>
+ let output_password =<br>
+ match output_password with<br>
+ | None -><br>
+ error (f_"-o rhv-upload: output password file was not specified, use ‘-op’ to point to a file which contains the password used to connect to the oVirt or RHV server")<br>
+ | Some op -> op in<br>
+ let os =<br>
+ match output_storage with<br>
+ | None -><br>
+ error (f_"-o rhv-upload: output storage was not specified, use ‘-os’");<br>
+ | Some os -> os in<br>
+ if qemu_boot then<br>
+ error_option_cannot_be_used_in_output_mode "rhv-upload" "--qemu-boot";<br>
+ let rhv_cafile =<br>
+ match rhv_cafile with<br>
+ | None -><br>
+ error (f_"-o rhv-upload: must use ‘--rhv-cafile’ to supply the path to the oVirt or RHV server’s ‘ca.pem’ file")<br>
+ | Some rhv_cafile -> rhv_cafile in<br>
+ Output_rhv_upload.output_rhv_upload output_alloc output_conn<br>
+ output_password os<br>
+ rhv_cafile rhv_direct,<br>
+ output_format, output_alloc<br>
+<br>
| `VDSM -><br>
if output_password <> None then<br>
error_option_cannot_be_used_in_output_mode "vdsm" "-op";<br>
diff --git a/v2v/embed.sh b/v2v/embed.sh<br>
new file mode 100755<br>
index 000000000..0a65cd428<br>
--- /dev/null<br>
+++ b/v2v/embed.sh<br>
@@ -0,0 +1,45 @@<br>
+#!/bin/bash -<br>
+# Embed code or other content into an OCaml file.<br>
+# Copyright (C) 2018 Red Hat Inc.<br>
+#<br>
+# This program is free software; you can redistribute it and/or modify<br>
+# it under the terms of the GNU General Public License as published by<br>
+# the Free Software Foundation; either version 2 of the License, or<br>
+# (at your option) any later version.<br>
+#<br>
+# This program is distributed in the hope that it will be useful,<br>
+# but WITHOUT ANY WARRANTY; without even the implied warranty of<br>
+# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the<br>
+# GNU General Public License for more details.<br>
+#<br>
+# You should have received a copy of the GNU General Public License<br>
+# along with this program; if not, <a href="https://maps.google.com/?q=write+to+the+Free&entry=gmail&source=g">write to the Free</a> Software<br>
+# Foundation, Inc., 51 Franklin Street, Fifth Floor, Boston, MA 02110-1301 USA.<br>
+<br>
+# Embed code or other content into an OCaml file.<br>
+#<br>
+# It is embedded into a string. As OCaml string literals have virtually<br>
+# no restrictions on length or content we only have to escape double<br>
+# quotes for backslash characters.<br>
+<br>
+if [ $# -ne 3 ]; then<br>
+ echo "embed.sh identifier input output"<br>
+ exit 1<br>
+fi<br>
+<br>
+ident="$1"<br>
+input="$2"<br>
+output="$3"<br>
+<br>
+rm -f "$output" "$output"-t<br>
+<br>
+exec >"$output"-t<br>
+<br>
+echo "(* Generated by embed.sh from $input *)"<br>
+echo<br>
+echo let "$ident" = '"'<br>
+sed -e 's/\(["\]\)/\\\1/g' < "$input"<br>
+echo '"'<br>
+<br>
+chmod -w "$output"-t<br>
+mv "$output"-t "$output"<br>
diff --git a/v2v/<a href="http://output_rhv_upload.ml" rel="noreferrer" target="_blank">output_rhv_upload.ml</a> b/v2v/<a href="http://output_rhv_upload.ml" rel="noreferrer" target="_blank">output_rhv_upload.ml</a><br>
new file mode 100644<br>
index 000000000..eecabd74d<br>
--- /dev/null<br>
+++ b/v2v/<a href="http://output_rhv_upload.ml" rel="noreferrer" target="_blank">output_rhv_upload.ml</a><br>
@@ -0,0 +1,330 @@<br>
+(* virt-v2v<br>
+ * Copyright (C) 2009-2018 Red Hat Inc.<br>
+ *<br>
+ * This program is free software; you can redistribute it and/or modify<br>
+ * it under the terms of the GNU General Public License as published by<br>
+ * the Free Software Foundation; either version 2 of the License, or<br>
+ * (at your option) any later version.<br>
+ *<br>
+ * This program is distributed in the hope that it will be useful,<br>
+ * but WITHOUT ANY WARRANTY; without even the implied warranty of<br>
+ * MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the<br>
+ * GNU General Public License for more details.<br>
+ *<br>
+ * You should have received a copy of the GNU General Public License along<br>
+ * with this program; if not, <a href="https://maps.google.com/?q=write+to+the+Free+&entry=gmail&source=g">write to the Free </a>Software Foundation, Inc.,<br>
+ * 51 Franklin Street, Fifth Floor, Boston, MA 02110-1301 USA.<br>
+ *)<br>
+<br>
+open Printf<br>
+open Unix<br>
+<br>
+open Std_utils<br>
+open Tools_utils<br>
+open Unix_utils<br>
+open Common_gettext.Gettext<br>
+<br>
+open Types<br>
+open Utils<br>
+<br>
+let python3 = "python3" (* Defined by PEP 394 *)<br>
+let pidfile_timeout = 30<br>
+let finalization_timeout = 5*60<br>
+<br>
+class output_rhv_upload output_alloc output_conn<br>
+ output_password output_storage<br>
+ rhv_cafile rhv_direct =<br>
+ (* Create a temporary directory which will be deleted on exit. *)<br>
+ let tmpdir =<br>
+ let base_dir = (open_guestfs ())#get_cachedir () in<br>
+ let t = Mkdtemp.temp_dir ~base_dir "rhvupload." in<br>
+ rmdir_on_exit t;<br>
+ t in<br>
+<br>
+ let diskid_file_of_id id = tmpdir // sprintf "diskid.%d" id in<br>
+<br>
+ (* Write the Python precheck, plugin and create VM to a temporary file. *)<br>
+ let precheck =<br>
+ let precheck = tmpdir // "rhv-upload-precheck.py" in<br>
+ with_open_out<br>
+ precheck<br>
+ (fun chan -> output_string chan Output_rhv_upload_precheck_source.code);<br>
+ precheck in<br>
+ let plugin =<br>
+ let plugin = tmpdir // "rhv-upload-plugin.py" in<br>
+ with_open_out<br>
+ plugin<br>
+ (fun chan -> output_string chan Output_rhv_upload_plugin_source.code);<br>
+ plugin in<br>
+ let createvm =<br>
+ let createvm = tmpdir // "rhv-upload-createvm.py" in<br>
+ with_open_out<br>
+ createvm<br>
+ (fun chan -> output_string chan Output_rhv_upload_createvm_source.code);<br>
+ createvm in<br>
+<br>
+ (* Is SELinux enabled and enforcing on the host? *)<br>
+ let have_selinux =<br>
+ 0 = Sys.command "getenforce 2>/dev/null | grep -isq Enforcing" in<br>
+<br>
+ (* Check that nbdkit is available and new enough. *)<br>
+ let error_unless_nbdkit_working () =<br>
+ if 0 <> Sys.command "nbdkit --version >/dev/null" then<br>
+ error (f_"nbdkit is not installed or not working. It is required to use ‘-o rhv-upload’. See \"OUTPUT TO RHV\" in the virt-v2v(1) manual.");<br>
+<br>
+ (* Check it's a new enough version. The latest features we<br>
+ * require are ‘--exit-with-parent’ and ‘--selinux-label’, both<br>
+ * added in 1.1.14. (We use 1.1.16 as the minimum here because<br>
+ * it also adds the selinux=yes|no flag in --dump-config).<br>
+ *)<br>
+ let lines = external_command "nbdkit --help" in<br>
+ let lines = String.concat " " lines in<br>
+ if String.find lines "exit-with-parent" == -1 ||<br>
+ String.find lines "selinux-label" == -1 then<br>
+ error (f_"nbdkit is not new enough, you need to upgrade to nbdkit ≥ 1.1.16")<br>
+ in<br>
+<br>
+ (* Check that the python3 plugin is installed and working<br>
+ * and can load the plugin script.<br>
+ *)<br>
+ let error_unless_nbdkit_python3_working () =<br>
+ let cmd = sprintf "nbdkit %s %s --dump-plugin >/dev/null"<br>
+ python3 (quote plugin) in<br>
+ if Sys.command cmd <> 0 then<br>
+ error (f_"nbdkit Python 3 plugin is not installed or not working. It is required if you want to use ‘-o rhv-upload’.<br>
+<br>
+See also \"OUTPUT TO RHV\" in the virt-v2v(1) manual.")<br>
+ in<br>
+<br>
+ (* Check that nbdkit was compiled with SELinux support (for the<br>
+ * --selinux-label option).<br>
+ *)<br>
+ let error_unless_nbdkit_compiled_with_selinux () =<br>
+ let lines = external_command "nbdkit --dump-config" in<br>
+ (* In nbdkit <= 1.1.15 the selinux attribute was not present<br>
+ * at all in --dump-config output so there was no way to tell.<br>
+ * Ignore this case because there will be an error later when<br>
+ * we try to use the --selinux-label parameter.<br>
+ *)<br>
+ if List.mem "selinux=no" (List.map String.trim lines) then<br>
+ error (f_"nbdkit was compiled without SELinux support. You will have to recompile nbdkit with libselinux-devel installed, or else set SELinux to Permissive mode while doing the conversion.")<br>
+ in<br>
+<br>
+ (* JSON parameters which are invariant between disks. *)<br>
+ let json_params = [<br>
+ "output_conn", JSON.String output_conn;<br>
+ "output_password", JSON.String output_password;<br>
+ "output_storage", JSON.String output_storage;<br>
+ "output_sparse", JSON.Bool (match output_alloc with<br>
+ | Sparse -> true<br>
+ | Preallocated -> false);<br>
+ "rhv_cafile", JSON.String rhv_cafile;<br>
+ "rhv_direct", JSON.Bool rhv_direct;<br>
+ ] in<br>
+<br>
+ (* nbdkit command line args which are invariant between disks. *)<br>
+ let nbdkit_args =<br>
+ let args = [<br>
+ "nbdkit";<br>
+<br>
+ "--foreground"; (* run in foreground *)<br>
+ "--exit-with-parent"; (* exit when virt-v2v exits *)<br>
+ "--newstyle"; (* use newstyle NBD protocol *)<br>
+ "--exportname"; "/";<br>
+<br>
+ "python3"; (* use the nbdkit Python 3 plugin *)<br>
+ plugin; (* Python plugin script *)<br>
+ ] in<br>
+ let args = if verbose () then args @ ["--verbose"] else args in<br>
+ let args =<br>
+ (* label the socket so qemu can open it *)<br>
+ if have_selinux then<br>
+ args @ ["--selinux-label"; "system_u:object_r:svirt_t:s0"]<br>
+ else args in<br>
+ args in<br>
+<br>
+object<br>
+ inherit output<br>
+<br>
+ method precheck () =<br>
+ error_unless_nbdkit_working ();<br>
+ error_unless_nbdkit_python3_working ();<br>
+ if have_selinux then<br>
+ error_unless_nbdkit_compiled_with_selinux ()<br>
+<br>
+ method as_options =<br>
+ "-o rhv-upload" ^<br>
+ (match output_alloc with<br>
+ | Sparse -> "" (* default, don't need to print it *)<br>
+ | Preallocated -> " -oa preallocated") ^<br>
+ sprintf " -oc %s -op %s -os %s"<br>
+ output_conn output_password output_storage<br>
+<br>
+ method supported_firmware = [ TargetBIOS ]<br>
+<br>
+ method prepare_targets source targets =<br>
+ let output_name = source.s_name in<br>
+ let json_params =<br>
+ ("output_name", JSON.String output_name) :: json_params in<br>
+<br>
+ (* Python code prechecks. These can't run in #precheck because<br>
+ * we need to know the name of the virtual machine.<br>
+ *)<br>
+ let json_param_file = tmpdir // "params.json" in<br>
+ with_open_out<br>
+ json_param_file<br>
+ (fun chan -> output_string chan (JSON.string_of_doc json_params));<br>
+ if run_command [ python3; precheck; json_param_file ] <> 0 then<br>
+ error (f_"failed server prechecks, see earlier errors");<br>
+<br>
+ (* Create an nbdkit instance for each disk and set the<br>
+ * target URI to point to the NBD socket.<br>
+ *)<br>
+ List.map (<br>
+ fun t -><br>
+ let id = t.target_overlay.ov_source.s_disk_id in<br>
+ let disk_name = sprintf "%s-%03d" output_name id in<br>
+ let json_params =<br>
+ ("disk_name", JSON.String disk_name) :: json_params in<br>
+<br>
+ let disk_format =<br>
+ match t.target_format with<br>
+ | ("raw" | "qcow2") as fmt -> fmt<br>
+ | _ -><br>
+ error (f_"rhv-upload: -of %s: Only output format ‘raw’ or ‘qcow2’ is supported. If the input is in a different format then force one of these output formats by adding either ‘-of raw’ or ‘-of qcow2’ on the command line.")<br>
+ t.target_format in<br>
+ let json_params =<br>
+ ("disk_format", JSON.String disk_format) :: json_params in<br>
+<br>
+ let disk_size = t.target_overlay.ov_virtual_size in<br>
+ let json_params =<br>
+ ("disk_size", JSON.Int64 disk_size) :: json_params in<br>
+<br>
+ (* Ask the plugin to write the disk ID to a special file. *)<br>
+ let diskid_file = diskid_file_of_id id in<br>
+ let json_params =<br>
+ ("diskid_file", JSON.String diskid_file) :: json_params in<br>
+<br>
+ (* Write the JSON parameters to a file. *)<br>
+ let json_param_file = tmpdir // sprintf "params%d.json" id in<br>
+ with_open_out<br>
+ json_param_file<br>
+ (fun chan -> output_string chan (JSON.string_of_doc json_params));<br>
+<br>
+ let sock = tmpdir // sprintf "nbdkit%d.sock" id in<br>
+ let pidfile = tmpdir // sprintf "nbdkit%d.pid" id in<br>
+<br>
+ (* Add common arguments to per-target arguments. *)<br>
+ let args =<br>
+ nbdkit_args @ [ "--pidfile"; pidfile;<br>
+ "--unix"; sock;<br>
+ sprintf "params=%s" json_param_file ] in<br>
+<br>
+ (* Print the full command we are about to run when debugging. *)<br>
+ if verbose () then (<br>
+ eprintf "running nbdkit:\n";<br>
+ List.iter (fun arg -> eprintf " %s" (quote arg)) args;<br>
+ prerr_newline ()<br>
+ );<br>
+<br>
+ (* Start an nbdkit instance in the background. By using<br>
+ * --exit-with-parent we don't have to worry about clean-up.<br>
+ *)<br>
+ let args = Array.of_list args in<br>
+ let pid = fork () in<br>
+ if pid = 0 then (<br>
+ (* Child process (nbdkit). *)<br>
+ execvp "nbdkit" args<br>
+ );<br>
+<br>
+ (* Wait for the pidfile to appear so we know that nbdkit<br>
+ * is listening for requests.<br>
+ *)<br>
+ if not (wait_for_file pidfile pidfile_timeout) then (<br>
+ if verbose () then<br>
+ error (f_"nbdkit did not start up. See previous debugging messages for problems.")<br>
+ else<br>
+ error (f_"nbdkit did not start up. There may be errors printed by nbdkit above.<br>
+<br>
+If the messages above are not sufficient to diagnose the problem then add the ‘virt-v2v -v -x’ options and examine the debugging output carefully.")<br>
+ );<br>
+<br>
+ if have_selinux then (<br>
+ (* Note that Unix domain sockets have both a file label and<br>
+ * a socket/process label. Using --selinux-label above<br>
+ * only set the socket label, but we must also set the file<br>
+ * label.<br>
+ *)<br>
+ ignore (<br>
+ run_command ["chcon"; "system_u:object_r:svirt_image_t:s0";<br>
+ sock]<br>
+ );<br>
+ );<br>
+ (* ... and the regular Unix permissions, in case qemu is<br>
+ * running as another user.<br>
+ *)<br>
+ chmod sock 0o777;<br>
+<br>
+ (* Tell ‘qemu-img convert’ to write to the nbd socket which is<br>
+ * connected to nbdkit.<br>
+ *)<br>
+ let json_params = [<br>
+ "file.driver", JSON.String "nbd";<br>
+ "file.path", JSON.String sock;<br>
+ "file.export", JSON.String "/";<br>
+ ] in<br>
+ let target_file =<br>
+ TargetURI ("json:" ^ JSON.string_of_doc json_params) in<br>
+ { t with target_file }<br>
+ ) targets<br>
+<br>
+ method create_metadata source targets _ guestcaps inspect target_firmware =<br>
+ (* Get the UUIDs of each disk image. These files are written<br>
+ * out by the nbdkit plugins on successful finalization of the<br>
+ * transfer.<br>
+ *)<br>
+ let nr_disks = List.length targets in<br>
+ let image_uuids =<br>
+ List.map (<br>
+ fun t -><br>
+ let id = t.target_overlay.ov_source.s_disk_id in<br>
+ let diskid_file = diskid_file_of_id id in<br>
+ if not (wait_for_file diskid_file finalization_timeout) then<br>
+ error (f_"transfer of disk %d/%d failed, see earlier error messages")<br>
+ (id+1) nr_disks;<br>
+ let diskid = read_whole_file diskid_file in<br>
+ diskid<br>
+ ) targets in<br>
+<br>
+ (* We don't have the storage domain UUID, but instead we write<br>
+ * in a magic value which the Python code (which can get it)<br>
+ * will substitute.<br>
+ *)<br>
+ let sd_uuid = "@SD_UUID@" in<br>
+<br>
+ (* The volume and VM UUIDs are made up. *)<br>
+ let vol_uuids = List.map (fun _ -> uuidgen ()) targets<br>
+ and vm_uuid = uuidgen () in<br>
+<br>
+ (* Create the metadata. *)<br>
+ let ovf =<br>
+ Create_ovf.create_ovf source targets guestcaps inspect<br>
+ output_alloc<br>
+ sd_uuid image_uuids vol_uuids vm_uuid<br>
+ OVirt in<br>
+ let ovf = DOM.doc_to_string ovf in<br>
+<br>
+ let json_param_file = tmpdir // "params.json" in<br>
+ with_open_out<br>
+ json_param_file<br>
+ (fun chan -> output_string chan (JSON.string_of_doc json_params));<br>
+<br>
+ let ovf_file = tmpdir // "vm.ovf" in<br>
+ with_open_out ovf_file (fun chan -> output_string chan ovf);<br>
+ if run_command [ python3; createvm; json_param_file; ovf_file ] <> 0 then<br>
+ error (f_"failed to create virtual machine, see earlier errors")<br>
+<br>
+end<br>
+<br>
+let output_rhv_upload = new output_rhv_upload<br>
+let () = Modules_list.register_output_module "rhv-upload"<br>
diff --git a/v2v/output_rhv_upload.mli b/v2v/output_rhv_upload.mli<br>
new file mode 100644<br>
index 000000000..3e7086f85<br>
--- /dev/null<br>
+++ b/v2v/output_rhv_upload.mli<br>
@@ -0,0 +1,27 @@<br>
+(* virt-v2v<br>
+ * Copyright (C) 2009-2018 Red Hat Inc.<br>
+ *<br>
+ * This program is free software; you can redistribute it and/or modify<br>
+ * it under the terms of the GNU General Public License as published by<br>
+ * the Free Software Foundation; either version 2 of the License, or<br>
+ * (at your option) any later version.<br>
+ *<br>
+ * This program is distributed in the hope that it will be useful,<br>
+ * but WITHOUT ANY WARRANTY; without even the implied warranty of<br>
+ * MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the<br>
+ * GNU General Public License for more details.<br>
+ *<br>
+ * You should have received a copy of the GNU General Public License along<br>
+ * with this program; if not, <a href="https://maps.google.com/?q=write+to+the+Free+&entry=gmail&source=g">write to the Free </a>Software Foundation, Inc.,<br>
+ * 51 Franklin Street, Fifth Floor, Boston, MA 02110-1301 USA.<br>
+ *)<br>
+<br>
+(** [-o rhv-upload] target. *)<br>
+<br>
+val output_rhv_upload : Types.output_allocation -> string -> string -><br>
+ string -> string -> bool -><br>
+ Types.output<br>
+(** [output_rhv_upload output_alloc output_conn output_password output_storage<br>
+ rhv_cafile rhv_direct]<br>
+ creates and returns a new {!Types.output} object specialized for writing<br>
+ output to oVirt or RHV directly via RHV APIs. *)<br>
diff --git a/v2v/output_rhv_upload_createvm_source.mli b/v2v/output_rhv_upload_createvm_source.mli<br>
new file mode 100644<br>
index 000000000..c1bafa15b<br>
--- /dev/null<br>
+++ b/v2v/output_rhv_upload_createvm_source.mli<br>
@@ -0,0 +1,19 @@<br>
+(* virt-v2v<br>
+ * Copyright (C) 2018 Red Hat Inc.<br>
+ *<br>
+ * This program is free software; you can redistribute it and/or modify<br>
+ * it under the terms of the GNU General Public License as published by<br>
+ * the Free Software Foundation; either version 2 of the License, or<br>
+ * (at your option) any later version.<br>
+ *<br>
+ * This program is distributed in the hope that it will be useful,<br>
+ * but WITHOUT ANY WARRANTY; without even the implied warranty of<br>
+ * MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the<br>
+ * GNU General Public License for more details.<br>
+ *<br>
+ * You should have received a copy of the GNU General Public License along<br>
+ * with this program; if not, <a href="https://maps.google.com/?q=write+to+the+Free+&entry=gmail&source=g">write to the Free </a>Software Foundation, Inc.,<br>
+ * 51 Franklin Street, Fifth Floor, Boston, MA 02110-1301 USA.<br>
+ *)<br>
+<br>
+val code : string<br>
diff --git a/v2v/output_rhv_upload_plugin_source.mli b/v2v/output_rhv_upload_plugin_source.mli<br>
new file mode 100644<br>
index 000000000..c1bafa15b<br>
--- /dev/null<br>
+++ b/v2v/output_rhv_upload_plugin_source.mli<br>
@@ -0,0 +1,19 @@<br>
+(* virt-v2v<br>
+ * Copyright (C) 2018 Red Hat Inc.<br>
+ *<br>
+ * This program is free software; you can redistribute it and/or modify<br>
+ * it under the terms of the GNU General Public License as published by<br>
+ * the Free Software Foundation; either version 2 of the License, or<br>
+ * (at your option) any later version.<br>
+ *<br>
+ * This program is distributed in the hope that it will be useful,<br>
+ * but WITHOUT ANY WARRANTY; without even the implied warranty of<br>
+ * MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the<br>
+ * GNU General Public License for more details.<br>
+ *<br>
+ * You should have received a copy of the GNU General Public License along<br>
+ * with this program; if not, <a href="https://maps.google.com/?q=write+to+the+Free+&entry=gmail&source=g">write to the Free </a>Software Foundation, Inc.,<br>
+ * 51 Franklin Street, Fifth Floor, Boston, MA 02110-1301 USA.<br>
+ *)<br>
+<br>
+val code : string<br>
diff --git a/v2v/output_rhv_upload_precheck_source.mli b/v2v/output_rhv_upload_precheck_source.mli<br>
new file mode 100644<br>
index 000000000..c1bafa15b<br>
--- /dev/null<br>
+++ b/v2v/output_rhv_upload_precheck_source.mli<br>
@@ -0,0 +1,19 @@<br>
+(* virt-v2v<br>
+ * Copyright (C) 2018 Red Hat Inc.<br>
+ *<br>
+ * This program is free software; you can redistribute it and/or modify<br>
+ * it under the terms of the GNU General Public License as published by<br>
+ * the Free Software Foundation; either version 2 of the License, or<br>
+ * (at your option) any later version.<br>
+ *<br>
+ * This program is distributed in the hope that it will be useful,<br>
+ * but WITHOUT ANY WARRANTY; without even the implied warranty of<br>
+ * MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the<br>
+ * GNU General Public License for more details.<br>
+ *<br>
+ * You should have received a copy of the GNU General Public License along<br>
+ * with this program; if not, <a href="https://maps.google.com/?q=write+to+the+Free+&entry=gmail&source=g">write to the Free </a>Software Foundation, Inc.,<br>
+ * 51 Franklin Street, Fifth Floor, Boston, MA 02110-1301 USA.<br>
+ *)<br>
+<br>
+val code : string<br>
diff --git a/v2v/rhv-upload-createvm.py b/v2v/rhv-upload-createvm.py<br>
new file mode 100644<br>
index 000000000..47c2574bb<br>
--- /dev/null<br>
+++ b/v2v/rhv-upload-createvm.py<br>
@@ -0,0 +1,85 @@<br>
+# -*- python -*-<br>
+# oVirt or RHV upload create VM used by ‘virt-v2v -o rhv-upload’<br>
+# Copyright (C) 2018 Red Hat Inc.<br>
+#<br>
+# This program is free software; you can redistribute it and/or modify<br>
+# it under the terms of the GNU General Public License as published by<br>
+# the Free Software Foundation; either version 2 of the License, or<br>
+# (at your option) any later version.<br>
+#<br>
+# This program is distributed in the hope that it will be useful,<br>
+# but WITHOUT ANY WARRANTY; without even the implied warranty of<br>
+# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the<br>
+# GNU General Public License for more details.<br>
+#<br>
+# You should have received a copy of the GNU General Public License along<br>
+# with this program; if not, <a href="https://maps.google.com/?q=write+to+the+Free&entry=gmail&source=g">write to the Free</a> Software Foundation, Inc.,<br>
+# 51 Franklin Street, Fifth Floor, Boston, MA 02110-1301 USA.<br>
+<br>
+import json<br>
+import logging<br>
+import ovirtsdk4 as sdk<br>
+import ovirtsdk4.types as types<br>
+import sys<br>
+import time<br>
+<br>
+from http.client import HTTPSConnection<br>
+from urllib.parse import urlparse<br>
+<br>
+# Parameters are passed in via a JSON doc from the OCaml code.<br>
+# Because this Python code ships embedded inside virt-v2v there<br>
+# is no formal API here.<br>
+params = None<br>
+ovf = None # OVF file<br>
+<br>
+if len(sys.argv) != 3:<br>
+ raise RuntimeError("incorrect number of parameters")<br>
+<br>
+# Parameters are passed in via a JSON document.<br>
+with open(sys.argv[1], 'r') as fp:<br>
+ params = json.load(fp)<br>
+<br>
+# What is passed in is a password file, read the actual password.<br>
+with open(params['output_password'], 'r') as fp:<br>
+ output_password = fp.read()<br>
+output_password = output_password.rstrip()<br>
+<br>
+# Read the OVF document.<br>
+with open(sys.argv[2], 'r') as fp:<br>
+ ovf = fp.read()<br>
+<br>
+# Parse out the username from the output_conn URL.<br>
+parsed = urlparse(params['output_conn'])<br>
+username = parsed.username or "admin@internal"<br>
+<br>
+# Connect to the server.<br>
+connection = sdk.Connection(<br>
+ url = params['output_conn'],<br>
+ username = username,<br>
+ password = output_password,<br>
+ ca_file = params['rhv_cafile'],<br>
+ log = logging.getLogger(),<br>
+ insecure = True, # XXX?<br></blockquote><div><br></div><div>ovirt-imageio authentication is based on the assumption that the </div><div>secret random url is passed from engine to the user via https.</div><div>if this access engine using clear text then yes it is bad :-)</div><div><br></div><div>Ondra, can you explain the semantics of incsecure=True?</div><div> </div><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex">
+)<br>
+<br>
+system_service = connection.system_service()<br>
+<br>
+# Get the storage domain UUID and substitute it into the OVF doc.<br>
+sds_service = system_service.storage_domains_service()<br>
+sd = sds_service.list(search=("name=%s" % params['output_storage']))[0]<br>
+sd_uuid = <a href="http://sd.id" rel="noreferrer" target="_blank">sd.id</a><br>
+<br>
+ovf.replace("@SD_UUID@", sd_uuid)<br>
+<br>
+vms_service = system_service.vms_service()<br>
+vm = vms_service.add(<br>
+ types.Vm(<br>
+ cluster=types.Cluster(name = "Default"), # XXX<br>
+ initialization=types.Initialization(<br>
+ configuration = types.Configuration(<br>
+ type = types.ConfigurationType.OVA,<br>
+ data = ovf,<br>
+ )<br>
+ )<br>
+ )<br>
+)<br>
diff --git a/v2v/rhv-upload-plugin.py b/v2v/rhv-upload-plugin.py<br>
new file mode 100644<br>
index 000000000..e65cb1c32<br>
--- /dev/null<br>
+++ b/v2v/rhv-upload-plugin.py<br>
@@ -0,0 +1,248 @@<br>
+# -*- python -*-<br>
+# oVirt or RHV upload nbdkit plugin used by ‘virt-v2v -o rhv-upload’<br>
+# Copyright (C) 2018 Red Hat Inc.<br>
+#<br>
+# This program is free software; you can redistribute it and/or modify<br>
+# it under the terms of the GNU General Public License as published by<br>
+# the Free Software Foundation; either version 2 of the License, or<br>
+# (at your option) any later version.<br>
+#<br>
+# This program is distributed in the hope that it will be useful,<br>
+# but WITHOUT ANY WARRANTY; without even the implied warranty of<br>
+# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the<br>
+# GNU General Public License for more details.<br>
+#<br>
+# You should have received a copy of the GNU General Public License along<br>
+# with this program; if not, <a href="https://maps.google.com/?q=write+to+the+Free&entry=gmail&source=g">write to the Free</a> Software Foundation, Inc.,<br>
+# 51 Franklin Street, Fifth Floor, Boston, MA 02110-1301 USA.<br>
+<br>
+import builtins<br>
+import json<br>
+import logging<br>
+import ovirtsdk4 as sdk<br>
+import ovirtsdk4.types as types<br>
+import ssl<br>
+import sys<br>
+import time<br>
+<br>
+from http.client import HTTPSConnection<br>
+from urllib.parse import urlparse<br>
+<br>
+# Timeout to wait for oVirt disks to change status, or the transfer<br>
+# object to finish initializing [seconds].<br>
+timeout = 5*60<br>
+<br>
+# Parameters are passed in via a JSON doc from the OCaml code.<br>
+# Because this Python code ships embedded inside virt-v2v there<br>
+# is no formal API here.<br>
+params = None<br>
+<br>
+def config(key, value):<br>
+ global params<br>
+<br>
+ if key == "params":<br>
+ with builtins.open(value, 'r') as fp:<br>
+ params = json.load(fp)<br>
+ else:<br>
+ raise RuntimeError("unknown configuration key '%s'" % key)<br>
+<br>
+def config_complete():<br>
+ global params<br>
+<br>
+ if params is None:<br>
+ raise RuntimeError("missing configuration parameters")<br>
+<br>
+def open(readonly):<br>
+ global params<br></blockquote><div><br></div><div>global is needed only when you try to bind something to the name, e.g.</div><div><br></div><div> params = {} </div><div><br></div><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex">
+<br>
+ # Parse out the username from the output_conn URL.<br>
+ parsed = urlparse(params['output_conn'])<br>
+ username = parsed.username or "admin@internal"<br>
+<br>
+ # Read the password from file.<br>
+ with builtins.open(params['output_password'], 'r') as fp:<br>
+ password = fp.read()<br>
+ password = password.rstrip()<br>
+<br>
+ # Connect to the server.<br>
+ connection = sdk.Connection(<br>
+ url = params['output_conn'],<br>
+ username = username,<br>
+ password = password,<br>
+ ca_file = params['rhv_cafile'],<br>
+ log = logging.getLogger(),<br>
+ insecure = True, # XXX?<br>
+ )<br>
+<br>
+ system_service = connection.system_service()<br>
+<br>
+ # Create the disk.<br>
+ disks_service = system_service.disks_service()<br>
+ if params['disk_format'] == "raw":<br>
+ disk_format = types.DiskFormat.RAW<br>
+ else:<br>
+ disk_format = types.DiskFormat.COW<br>
+ disk = disks_service.add(<br>
+ disk = types.Disk(<br>
+ name = params['disk_name'],<br>
+ description = "Uploaded by virt-v2v",<br>
+ format = disk_format,<br>
+ provisioned_size = params['disk_size'],<br>
+ sparse = params['output_sparse'],<br>
+ storage_domains = [<br>
+ types.StorageDomain(<br>
+ name = params['output_storage'],<br>
+ )<br>
+ ],<br>
+ )<br>
+ )<br>
+<br>
+ # Wait till the disk is up, as the transfer can't start if the<br>
+ # disk is locked:<br>
+ disk_service = disks_service.disk_service(<a href="http://disk.id" rel="noreferrer" target="_blank">disk.id</a>)<br>
+<br>
+ endt = time.time() + timeout<br>
+ while True:<br>
+ time.sleep(5)<br>
+ disk = disk_service.get()<br>
+ if disk.status == types.DiskStatus.OK:<br>
+ break<br>
+ if time.time() > endt:<br>
+ raise RuntimeError("timed out waiting for disk to become unlocked")<br>
+<br>
+ # Get a reference to the transfer service.<br>
+ transfers_service = system_service.image_transfers_service()<br>
+<br>
+ # Create a new image transfer.<br>
+ transfer = transfers_service.add(<br>
+ types.ImageTransfer(<br>
+ image = types.Image(<br>
+ id = <a href="http://disk.id" rel="noreferrer" target="_blank">disk.id</a><br>
+ )<br>
+ )<br>
+ )<br>
+<br>
+ # Get a reference to the created transfer service.<br>
+ transfer_service = transfers_service.image_transfer_service(<a href="http://transfer.id" rel="noreferrer" target="_blank">transfer.id</a>)<br>
+<br>
+ # After adding a new transfer for the disk, the transfer's status<br>
+ # will be INITIALIZING. Wait until the init phase is over. The<br>
+ # actual transfer can start when its status is "Transferring".<br>
+ endt = time.time() + timeout<br>
+ while True:<br>
+ time.sleep(5)<br>
+ transfer = transfer_service.get()<br>
+ if transfer.phase != types.ImageTransferPhase.INITIALIZING:<br>
+ break<br>
+ if time.time() > endt:<br>
+ raise RuntimeError("timed out waiting for transfer status != INITIALIZING")<br>
+<br>
+ # Now we have permission to start the transfer.<br>
+ if params['rhv_direct']:<br>
+ if transfer.transfer_url is None:<br>
+ raise RuntimeError("direct upload to host not supported, requires ovirt-engine >= 4.2 and only works when virt-v2v is run within the oVirt/RHV environment, eg. on an ovirt node.")<br>
+ destination_url = urlparse(transfer.transfer_url)<br>
+ else:<br>
+ destination_url = urlparse(transfer.proxy_url)<br>
+<br>
+ context = ssl.create_default_context()<br>
+ context.load_verify_locations(cafile = params['rhv_cafile'])<br>
+<br>
+ http = HTTPSConnection(<br>
+ destination_url.hostname,<br>
+ destination_url.port,<br>
+ context = context<br>
+ )<br>
+<br>
+ # Save everything we need to make requests in the handle.<br>
+ return {<br>
+ 'connection': connection,<br>
+ 'disk': disk,<br>
+ 'disk_service': disk_service,<br>
+ 'failed': False,<br>
+ 'http': http,<br>
+ 'path': destination_url.path,<br>
+ 'wrreqs': 0,<br>
+ 'transfer': transfer,<br>
+ 'transfer_service': transfer_service,<br>
+ }<br>
+<br>
+def get_size(h):<br>
+ global params<br>
+<br>
+ return params['disk_size']<br>
+<br>
+def pread(h, count, offset):<br>
+ http = h['http']<br>
+ transfer=h['transfer']<br>
+ transfer_service=h['transfer_service']<br>
+<br>
+ http.putrequest("GET", h['path'])<br>
+ http.putheader("Authorization", transfer.signed_ticket)<br>
+ http.putheader("Range", str(offset) + "-" + str(count+offset-1))<br></blockquote><div><br></div><div>We use "bytes=start-end"</div><div><br></div><div>See <a href="https://github.com/oVirt/ovirt-imageio/blob/master/daemon/test/server_test.py#L472">https://github.com/oVirt/ovirt-imageio/blob/master/daemon/test/server_test.py#L472</a></div><div><br></div><div>Also formatting cab be nicer and more consistent like this:</div><div><br></div><div> "bytes=%d-%d" % (offset, offset + count - 1)</div><div><br></div><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex">
+ http.endheaders()<br>
+<br>
+ r = http.getresponse()<br>
+ if r.status != 200:<br>
+ h['transfer_service'].pause()<br>
+ h['failed'] = True<br>
+ raise RuntimeError("could not read sector (%d, %d): %d: %s" %<br>
+ (offset, count, r.status, r.reason))<br>
+ return r.read()<br>
+<br>
+def pwrite(h, buf, offset):<br>
+ h['wrreqs'] += 1<br>
+ count = len(buf)<br>
+<br>
+ http = h['http']<br>
+ transfer=h['transfer']<br>
+ transfer_service=h['transfer_service']<br>
+<br>
+ http.putrequest("PUT", h['path'])<br>
+ http.putheader("Authorization", transfer.signed_ticket)<br>
+ http.putheader("Range", str(offset) + "-" + str(count+offset-1))<br></blockquote><div><br></div><div>Range is used only for GET, for PUT we use Content-Range</div><div>Current code will always write to 0-len(buf).</div><div><br></div><div>We use only the start value from the content range, so you can just send:</div><div><br></div><div> http.putheader("Content-Range", "bytes %d-*/*" % offset)</div><div><br></div><div>See <a href="https://github.com/oVirt/ovirt-imageio/blob/master/daemon/test/server_test.py#L393">https://github.com/oVirt/ovirt-imageio/blob/master/daemon/test/server_test.py#L393</a></div><div><br></div><div>I wonder if we should warn about unused Range header in PUT request.</div><div> </div><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex">
+ http.putheader("Content-Length", str(count))<br>
+ http.endheaders()<br>
+ http.send(buf)<br>
+<br>
+ r = http.getresponse()<br>
+ if r.status != 200:<br>
+ h['transfer_service'].pause()<br>
+ h['failed'] = True<br>
+ raise RuntimeError("could not write sector (%d, %d): %d: %s" %<br>
+ (offset, count, r.status, r.reason))<br>
+<br>
+# qemu-img convert starts by trying to zero/trim the whole device.<br>
+# Since we've just created a new disk it's safe to ignore these<br>
+# requests when they are the very first requests on the handle.<br>
+# After that we must emulate them with writes.</blockquote><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex">
+def zero(h, count, offset, may_trim):<br>
+ if h['wrreqs'] > 0:<br>
+ buf = bytearray(count)<br></blockquote><div><br></div><div>Can count be a large number (e.g. 1G)?</div><div><br></div><div>If the value is not limited in the caller, this should create reasonable</div><div>sized buffer and perform multiple writes </div><div><br></div><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex">
+ pwrite(h, buf, count)<br>
+<br>
+def close(h):<br>
+ global params<br>
+<br>
+ http = h['http']<br>
+ connection = h['connection']<br>
+<br>
+ http.close()<br>
+<br>
+ # If we didn't fail, then finalize the transfer.<br>
+ if not h['failed']:<br>
+ disk = h['disk']<br>
+ transfer_service=h['transfer_service']<br>
+<br>
+ transfer_service.finalize()<br>
+<br>
+ # Write the disk ID file. Only do this on successful completion.<br>
+ with builtins.open(params['diskid_file'], 'w') as fp:<br>
+ fp.write(<a href="http://disk.id" rel="noreferrer" target="_blank">disk.id</a>)<br>
+<br>
+ # Otherwise if we did fail then we should delete the disk.<br>
+ else:<br>
+ disk_service = h['disk_service']<br>
+ disk_service.remove()<br>
+<br>
+ connection.close()<br>
diff --git a/v2v/rhv-upload-precheck.py b/v2v/rhv-upload-precheck.py<br>
new file mode 100644<br>
index 000000000..fc060be41<br>
--- /dev/null<br>
+++ b/v2v/rhv-upload-precheck.py<br>
@@ -0,0 +1,72 @@<br>
+# -*- python -*-<br>
+# oVirt or RHV pre-upload checks used by ‘virt-v2v -o rhv-upload’<br>
+# Copyright (C) 2018 Red Hat Inc.<br>
+#<br>
+# This program is free software; you can redistribute it and/or modify<br>
+# it under the terms of the GNU General Public License as published by<br>
+# the Free Software Foundation; either version 2 of the License, or<br>
+# (at your option) any later version.<br>
+#<br>
+# This program is distributed in the hope that it will be useful,<br>
+# but WITHOUT ANY WARRANTY; without even the implied warranty of<br>
+# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the<br>
+# GNU General Public License for more details.<br>
+#<br>
+# You should have received a copy of the GNU General Public License along<br>
+# with this program; if not, <a href="https://maps.google.com/?q=write+to+the+Free&entry=gmail&source=g">write to the Free</a> Software Foundation, Inc.,<br>
+# 51 Franklin Street, Fifth Floor, Boston, MA 02110-1301 USA.<br>
+<br>
+import json<br>
+import logging<br>
+import ovirtsdk4 as sdk<br>
+import ovirtsdk4.types as types<br>
+import sys<br>
+import time<br>
+<br>
+from http.client import HTTPSConnection<br>
+from urllib.parse import urlparse<br>
+<br>
+# Parameters are passed in via a JSON doc from the OCaml code.<br>
+# Because this Python code ships embedded inside virt-v2v there<br>
+# is no formal API here.<br>
+params = None<br>
+<br>
+if len(sys.argv) != 2:<br>
+ raise RuntimeError("incorrect number of parameters")<br>
+<br>
+# Parameters are passed in via a JSON document.<br>
+with open(sys.argv[1], 'r') as fp:<br>
+ params = json.load(fp)<br>
+<br>
+# What is passed in is a password file, read the actual password.<br>
+with open(params['output_password'], 'r') as fp:<br>
+ output_password = fp.read()<br>
+output_password = output_password.rstrip()<br>
+<br>
+# Parse out the username from the output_conn URL.<br>
+parsed = urlparse(params['output_conn'])<br>
+username = parsed.username or "admin@internal"<br>
+<br>
+# Connect to the server.<br>
+connection = sdk.Connection(<br>
+ url = params['output_conn'],<br>
+ username = username,<br>
+ password = output_password,<br>
+ ca_file = params['rhv_cafile'],<br>
+ log = logging.getLogger(),<br>
+ insecure = True, # XXX?<br>
+)<br>
+<br>
+system_service = connection.system_service()<br>
+<br>
+# Find if a virtual machine already exists with that name.<br>
+vms_service = system_service.vms_service()<br>
+vms = vms_service.list(<br>
+ search = ("name=%s" % params['output_name']),<br>
+)<br>
+if len(vms) > 0:<br>
+ vm = vms[0]<br>
+ raise RuntimeError("VM already exists with name ‘%s’, id ‘%s’" %<br>
+ (params['output_name'], <a href="http://vm.id" rel="noreferrer" target="_blank">vm.id</a>))<br>
+<br>
+# Otherwise everything is OK, exit with no error.<br>
diff --git a/v2v/virt-v2v.pod b/v2v/virt-v2v.pod<br>
index d51e7ed2f..60cf682fd 100644<br>
--- a/v2v/virt-v2v.pod<br>
+++ b/v2v/virt-v2v.pod<br>
@@ -6,15 +6,18 @@ virt-v2v - Convert a guest to use KVM<br>
<br>
virt-v2v -ic vpx://<a href="http://vcenter.example.com/Datacenter/esxi" rel="noreferrer" target="_blank">vcenter.example.com/Datacenter/esxi</a> vmware_guest<br>
<br>
- virt-v2v -ic vpx://<a href="http://vcenter.example.com/Datacenter/esxi" rel="noreferrer" target="_blank">vcenter.example.com/Datacenter/esxi</a> vmware_guest \<br>
- -o rhv -os rhv.nfs:/export_domain --network ovirtmgmt<br>
-<br>
virt-v2v -i libvirtxml guest-domain.xml -o local -os /var/tmp<br>
<br>
virt-v2v -i disk disk.img -o local -os /var/tmp<br>
<br>
virt-v2v -i disk disk.img -o glance<br>
<br>
+ virt-v2v -ic vpx://<a href="http://vcenter.example.com/Datacenter/esxi" rel="noreferrer" target="_blank">vcenter.example.com/Datacenter/esxi</a> vmware_guest \<br>
+ -o rhv-upload -oc <a href="https://ovirt-engine.example.com/ovirt-engine/api" rel="noreferrer" target="_blank">https://ovirt-engine.example.com/ovirt-engine/api</a> \<br>
+ -os ovirt-data -op /tmp/ovirt-admin-password \<br>
+ --rhv-cafile /tmp/ca.pem --rhv-direct \<br>
+ --network ovirtmgmt<br>
+<br>
virt-v2v -ic qemu:///system qemu_guest --in-place<br>
<br>
=head1 DESCRIPTION<br>
@@ -42,9 +45,9 @@ libguestfs E<ge> 1.28.<br>
Xen ───▶│ -i libvirt ──▶ │ │ │ (default) │<br>
... ───▶│ (default) │ │ │ ──┐ └────────────┘<br>
└────────────┘ │ │ ─┐└──────▶ -o glance<br>
- -i libvirtxml ─────────▶ │ │ ┐└─────────▶ -o rhv<br>
- -i vmx ────────────────▶ │ │ └──────────▶ -o vdsm<br>
- └────────────┘<br>
+ -i libvirtxml ─────────▶ │ │ ┐├─────────▶ -o rhv<br>
+ -i vmx ────────────────▶ │ │ │└─────────▶ -o vdsm<br>
+ └────────────┘ └──────────▶ -o rhv-upload<br>
<br>
Virt-v2v has a number of possible input and output modes, selected<br>
using the I<-i> and I<-o> options. Only one input and output mode can<br>
@@ -103,20 +106,18 @@ For more information see L</INPUT FROM VMWARE VCENTER SERVER> below.<br>
=head2 Convert from VMware to RHV/oVirt<br>
<br>
This is the same as the previous example, except you want to send the<br>
-guest to a RHV-M Export Storage Domain which is located remotely<br>
-(over NFS) at C<rhv.nfs:/export_domain>. If you are unclear about<br>
-the location of the Export Storage Domain you should check the<br>
-settings on your RHV-M management console. Guest network<br>
+guest to a RHV Data Domain using the RHV REST API. Guest network<br>
interface(s) are connected to the target network called C<ovirtmgmt>.<br>
<br>
virt-v2v -ic vpx://<a href="http://vcenter.example.com/Datacenter/esxi" rel="noreferrer" target="_blank">vcenter.example.com/Datacenter/esxi</a> vmware_guest \<br>
- -o rhv -os rhv.nfs:/export_domain --network ovirtmgmt<br>
+ -o rhv-upload -oc <a href="https://ovirt-engine.example.com/ovirt-engine/api" rel="noreferrer" target="_blank">https://ovirt-engine.example.com/ovirt-engine/api</a> \<br>
+ -os ovirt-data -op /tmp/ovirt-admin-password \<br>
+ --rhv-cafile /tmp/ca.pem --rhv-direct \<br>
+ --network ovirtmgmt<br>
<br>
In this case the host running virt-v2v acts as a B<conversion server>.<br>
<br>
-Note that after conversion, the guest will appear in the RHV-M Export<br>
-Storage Domain, from where you will need to import it using the RHV-M<br>
-user interface. (See L</OUTPUT TO RHV>).<br>
+For more information see L</OUTPUT TO RHV> below.<br>
<br>
=head2 Convert from ESXi hypervisor over SSH to local libvirt<br>
<br>
@@ -509,6 +510,10 @@ written.<br>
<br>
This is the same as I<-o rhv>.<br>
<br>
+=item B<-o> B<ovirt-upload><br>
+<br>
+This is the same as I<-o rhv-upload>.<br>
+<br>
=item B<-o> B<qemu><br>
<br>
Set the output method to I<qemu>.<br>
@@ -533,6 +538,16 @@ I<-os> parameter must also be used to specify the location of the<br>
Export Storage Domain. Note this does not actually import the guest<br>
into RHV. You have to do that manually later using the UI.<br>
<br>
+See L</OUTPUT TO RHV (OLD METHOD)> below.<br>
+<br>
+=item B<-o> B<rhv-upload><br>
+<br>
+Set the output method to I<rhv-upload>.<br>
+<br>
+The converted guest is written directly to a RHV Data Domain.<br>
+This is a faster method than I<-o rhv>, but requires oVirt<br>
+or RHV E<ge> 4.2.<br>
+<br>
See L</OUTPUT TO RHV> below.<br>
<br>
=item B<-o> B<vdsm><br>
@@ -627,6 +642,21 @@ virt-v2v finishes.<br>
<br>
This disables progress bars and other unnecessary output.<br>
<br>
+=item B<--rhv-cafile> F<ca.pem><br>
+<br>
+For I<-o rhv-upload> (L<OUTPUT TO RHV>) only, the F<ca.pem> file<br>
+(Certificate Authority), copied from F</etc/pki/ovirt-engine/ca.pem><br>
+on the oVirt engine.<br>
+<br>
+=item B<--rhv-direct><br>
+<br>
+For I<-o rhv-upload> (L<OUTPUT TO RHV>) only, if this option is given<br>
+then virt-v2v will attempt to directly upload the disk to the oVirt<br>
+node, otherwise it will proxy the upload through the oVirt engine.<br>
+Direct upload requires that you have network access to the oVirt<br>
+nodes. Non-direct upload is slightly slower but should work in all<br>
+situations.<br>
+<br>
=item B<--root ask><br>
<br>
=item B<--root single><br>
@@ -1870,6 +1900,53 @@ Define the final guest in libvirt:<br>
<br>
=head1 OUTPUT TO RHV<br>
<br>
+This new method to upload guests to oVirt or RHV directly via the REST<br>
+API requires oVirt/RHV E<ge> 4.2.<br>
+<br>
+You need to specify I<-o rhv-upload> as well as the following extra<br>
+parameters:<br>
+<br>
+=over 4<br>
+<br>
+=item I<-oc> C<<a href="https://ovirt-engine.example.com/ovirt-engine/api" rel="noreferrer" target="_blank">https://ovirt-engine.example.com/ovirt-engine/api</a>><br>
+<br>
+The URL of the REST API which is usually the server name with<br>
+C</ovirt-engine/api> appended, but might be different if you installed<br>
+oVirt Engine on a different path.<br>
+<br>
+You can optionally add a username and port number to the URL. If the<br>
+username is not specified then virt-v2v defaults to using<br>
+C<admin@internal> which is the typical superuser account for oVirt<br>
+instances.<br>
+<br>
+=item I<-op> F<password-file><br>
+<br>
+A file containing a password to be used when connecting to the oVirt<br>
+engine. Note the file should contain the whole password, B<without<br>
+any trailing newline>, and for security the file should have mode<br>
+C<0600> so that others cannot read it.<br>
+<br>
+=item I<-os> C<ovirt-data><br>
+<br>
+The storage domain.<br>
+<br>
+=item I<--rhv-cafile> F<ca.pem><br>
+<br>
+The F<ca.pem> file (Certificate Authority), copied from<br>
+F</etc/pki/ovirt-engine/ca.pem> on the oVirt engine.<br>
+<br>
+=item I<--rhv-direct><br>
+<br>
+If this option is given then virt-v2v will attempt to directly upload<br>
+the disk to the oVirt node, otherwise it will proxy the upload through<br>
+the oVirt engine. Direct upload requires that you have network access<br>
+to the oVirt nodes. Non-direct upload is slightly slower but should<br>
+work in all situations.<br>
+<br>
+=back<br>
+<br>
+=head1 OUTPUT TO RHV (OLD METHOD)<br>
+<br>
This section only applies to the I<-o rhv> output mode. If you use<br>
virt-v2v from the RHV-M user interface, then behind the scenes the<br>
import is managed by VDSM using the I<-o vdsm> output mode (which end<br>
--<br>
2.13.2<br>
<br>
_______________________________________________<br>
Libguestfs mailing list<br>
<a href="mailto:Libguestfs@redhat.com" target="_blank">Libguestfs@redhat.com</a><br>
<a href="https://www.redhat.com/mailman/listinfo/libguestfs" rel="noreferrer" target="_blank">https://www.redhat.com/mailman/listinfo/libguestfs</a></blockquote></div></div>