[Libguestfs] [PATCH v8] v2v: Add -o rhv-upload output mode (RHBZ#1557273).
Tomáš Golembiovský
tgolembi at redhat.com
Fri Apr 6 18:28:29 UTC 2018
Hi,
few small comments below, but generally LGTM.
On Thu, 5 Apr 2018 14:26:07 +0100
"Richard W.M. Jones" <rjones at redhat.com> wrote:
> This adds a new output mode to virt-v2v. virt-v2v -o rhv-upload
> streams images directly to an oVirt or RHV >= 4 Data Domain using the
> oVirt SDK v4. It is more efficient than -o rhv because it does not
> need to go via the Export Storage Domain, and is possible for humans
> to use unlike -o vdsm.
>
> The implementation uses the Python SDK (‘ovirtsdk4’ module). An
> nbdkit Python 3 plugin translates NBD calls from qemu into HTTPS
> requests to oVirt via the SDK.
> ---
> .gitignore | 3 +
> TODO | 27 ++
> v2v/Makefile.am | 30 +-
> v2v/cmdline.ml | 41 +++
> v2v/embed.sh | 45 +++
> v2v/output_rhv_upload.ml | 390 ++++++++++++++++++++++++++
> v2v/output_rhv_upload.mli | 33 +++
> v2v/output_rhv_upload_createvm_source.mli | 19 ++
> v2v/output_rhv_upload_plugin_source.mli | 19 ++
> v2v/output_rhv_upload_precheck_source.mli | 19 ++
> v2v/rhv-upload-createvm.py | 86 ++++++
> v2v/rhv-upload-plugin.py | 438 ++++++++++++++++++++++++++++++
> v2v/rhv-upload-precheck.py | 73 +++++
> v2v/test-v2v-o-rhv-upload-oo-query.sh | 38 +++
> v2v/test-v2v-python-syntax.sh | 45 +++
> v2v/virt-v2v.pod | 138 ++++++++--
> 16 files changed, 1426 insertions(+), 18 deletions(-)
>
> diff --git a/.gitignore b/.gitignore
> index d72447d1d..211376eef 100644
> --- a/.gitignore
> +++ b/.gitignore
> @@ -654,6 +654,9 @@ Makefile.in
> /utils/qemu-speed-test/qemu-speed-test
> /v2v/.depend
> /v2v/oUnit-*
> +/v2v/output_rhv_upload_createvm_source.ml
> +/v2v/output_rhv_upload_plugin_source.ml
> +/v2v/output_rhv_upload_precheck_source.ml
> /v2v/real-*.d/
> /v2v/real-*.img
> /v2v/real-*.xml
> diff --git a/TODO b/TODO
> index 2e37ce67c..d196a3f6b 100644
> --- a/TODO
> +++ b/TODO
> @@ -570,3 +570,30 @@ Subsecond handling in virt-diff, virt-ls
>
> Handle nanoseconds properly. You should be able to specify them on
> the command line and display them.
> +
> +virt-v2v -o rhv-upload
> +----------------------
> +
> +* Set or disable the ticket timeout. The default is going to be
> + increased (from current 60 seconds), so maybe we won't have to
> + set it. See also:
> + https://bugzilla.redhat.com/show_bug.cgi?id=1563278
> +
> +* qcow2 cannot be supported yet because there is not yet any
> + concept in imageio of read+write handles.
> + https://bugzilla.redhat.com/show_bug.cgi?id=1563299
> +
> +* preallocated cannot be supported yet because imageio doesn't
> + know how to zero the image efficiently, instead it runs an
> + fallocate process which writes to every block and that takes
> + many minutes.
> +
> +* Really check what insecure/rhv_cafile do and implement it correctly.
> +
> +* Measure and resolve performance problems.
> +
> +* Allocated image size is unknown for v2v uploads, but imageio needs
> + to know it. We pass initial_size == provisioned_size == virtual size.
> + That can't be fixed from the v2v side.
> +
> +* There are unresolved issues about how to clean up disks on failure.
> diff --git a/v2v/Makefile.am b/v2v/Makefile.am
> index 6e71cae3c..f36731750 100644
> --- a/v2v/Makefile.am
> +++ b/v2v/Makefile.am
> @@ -22,12 +22,19 @@ generator_built = \
> uefi.mli
>
> BUILT_SOURCES = \
> - $(generator_built)
> + $(generator_built) \
> + output_rhv_upload_createvm_source.ml \
> + output_rhv_upload_plugin_source.ml \
> + output_rhv_upload_precheck_source.ml
>
> EXTRA_DIST = \
> $(SOURCES_MLI) $(SOURCES_ML) $(SOURCES_C) \
> copy_to_local.ml \
> copy_to_local.mli \
> + embed-code.sh \
> + rhv-upload-createvm.py \
> + rhv-upload-plugin.py \
> + rhv-upload-precheck.py \
> v2v_slow_unit_tests.ml \
> v2v-slow-unit-tests.sh \
> v2v_unit_tests.ml \
> @@ -64,6 +71,10 @@ SOURCES_MLI = \
> output_null.mli \
> output_qemu.mli \
> output_rhv.mli \
> + output_rhv_upload.mli \
> + output_rhv_upload_createvm_source.mli \
> + output_rhv_upload_plugin_source.mli \
> + output_rhv_upload_precheck_source.mli \
> output_vdsm.mli \
> parse_ovf_from_ova.mli \
> parse_libvirt_xml.mli \
> @@ -116,6 +127,10 @@ SOURCES_ML = \
> output_local.ml \
> output_qemu.ml \
> output_rhv.ml \
> + output_rhv_upload_createvm_source.ml \
> + output_rhv_upload_plugin_source.ml \
> + output_rhv_upload_precheck_source.ml \
> + output_rhv_upload.ml \
> output_vdsm.ml \
> inspect_source.ml \
> target_bus_assignment.ml \
> @@ -126,6 +141,15 @@ SOURCES_C = \
> libvirt_utils-c.c \
> qemuopts-c.c
>
> +# These files are generated and contain rhv-upload-*.py embedded as an
> +# OCaml string.
> +output_rhv_upload_createvm_source.ml: rhv-upload-createvm.py
> + ./embed.sh code $^ $@
> +output_rhv_upload_plugin_source.ml: rhv-upload-plugin.py
> + ./embed.sh code $^ $@
> +output_rhv_upload_precheck_source.ml: rhv-upload-precheck.py
> + ./embed.sh code $^ $@
> +
> if HAVE_OCAML
>
> bin_PROGRAMS = virt-v2v virt-v2v-copy-to-local
> @@ -295,6 +319,7 @@ TESTS_ENVIRONMENT = $(top_builddir)/run --test
>
> TESTS = \
> test-v2v-docs.sh \
> + test-v2v-python-syntax.sh \
> test-v2v-i-ova-bad-sha1.sh \
> test-v2v-i-ova-bad-sha256.sh \
> test-v2v-i-ova-formats.sh \
> @@ -307,6 +332,7 @@ TESTS = \
> test-v2v-i-ova-two-disks.sh \
> test-v2v-i-vmx.sh \
> test-v2v-it-vddk-io-query.sh \
> + test-v2v-o-rhv-upload-oo-query.sh \
> test-v2v-o-vdsm-oo-query.sh \
> test-v2v-bad-networks-and-bridges.sh
>
> @@ -485,6 +511,7 @@ EXTRA_DIST += \
> test-v2v-o-qemu.sh \
> test-v2v-o-rhv.ovf.expected \
> test-v2v-o-rhv.sh \
> + test-v2v-o-rhv-upload-oo-query.sh \
> test-v2v-o-vdsm-oo-query.sh \
> test-v2v-o-vdsm-options.ovf.expected \
> test-v2v-o-vdsm-options.sh \
> @@ -494,6 +521,7 @@ EXTRA_DIST += \
> test-v2v-print-source.expected \
> test-v2v-print-source.sh \
> test-v2v-print-source.xml \
> + test-v2v-python-syntax.sh \
> test-v2v-conversion-of.sh \
> test-v2v-sound.sh \
> test-v2v-sound.xml \
> diff --git a/v2v/cmdline.ml b/v2v/cmdline.ml
> index b36ec9c5e..785e91ef9 100644
> --- a/v2v/cmdline.ml
> +++ b/v2v/cmdline.ml
> @@ -135,6 +135,8 @@ let parse_cmdline () =
> | "disk" | "local" -> output_mode := `Local
> | "null" -> output_mode := `Null
> | "ovirt" | "rhv" | "rhev" -> output_mode := `RHV
> + | "ovirt-upload" | "ovirt_upload" | "rhv-upload" | "rhv_upload" ->
> + output_mode := `RHV_Upload
> | "qemu" -> output_mode := `QEmu
> | "vdsm" -> output_mode := `VDSM
> | s ->
> @@ -397,6 +399,16 @@ read the man page virt-v2v(1).
> | `Null -> no_options (); `Null
> | `RHV -> no_options (); `RHV
> | `QEmu -> no_options (); `QEmu
> + | `RHV_Upload ->
> + if is_query then (
> + Output_rhv_upload.print_output_options ();
> + exit 0
> + )
> + else (
> + let rhv_options =
> + Output_rhv_upload.parse_output_options output_options in
> + `RHV_Upload rhv_options
> + )
> | `VDSM ->
> if is_query then (
> Output_vdsm.print_output_options ();
> @@ -579,6 +591,35 @@ read the man page virt-v2v(1).
> Output_rhv.output_rhv os output_alloc,
> output_format, output_alloc
>
> + | `RHV_Upload rhv_options ->
> + let output_conn =
> + match output_conn with
> + | None ->
> + error (f_"-o rhv-upload: use ‘-oc’ to point to the oVirt or RHV server REST API URL, which is usually https://servername/ovirt-engine/api")
> + | Some oc -> oc in
> + (* Output format / sparse must currently be raw+sparse. We can
> + * change this in future. See TODO file for details. XXX
> + *)
> + if output_alloc <> Sparse || output_format <> Some "raw" then
> + error (f_"-o rhv-upload: currently you must use ‘-of raw’ and you cannot use ‘-oa preallocated’ with this output mode. These restrictions will be loosened in a future version.");
> + (* In theory we could make the password optional in future. *)
> + let output_password =
> + match output_password with
> + | None ->
> + error (f_"-o rhv-upload: output password file was not specified, use ‘-op’ to point to a file which contains the password used to connect to the oVirt or RHV server")
> + | Some op -> op in
> + let os =
> + match output_storage with
> + | None ->
> + error (f_"-o rhv-upload: output storage was not specified, use ‘-os’");
> + | Some os -> os in
> + if qemu_boot then
> + error_option_cannot_be_used_in_output_mode "rhv-upload" "--qemu-boot";
> + Output_rhv_upload.output_rhv_upload output_alloc output_conn
> + output_password os
> + rhv_options,
> + output_format, output_alloc
> +
> | `VDSM vdsm_options ->
> if output_password <> None then
> error_option_cannot_be_used_in_output_mode "vdsm" "-op";
> diff --git a/v2v/embed.sh b/v2v/embed.sh
> new file mode 100755
> index 000000000..0a65cd428
> --- /dev/null
> +++ b/v2v/embed.sh
> @@ -0,0 +1,45 @@
> +#!/bin/bash -
> +# Embed code or other content into an OCaml file.
> +# Copyright (C) 2018 Red Hat Inc.
> +#
> +# This program is free software; you can redistribute it and/or modify
> +# it under the terms of the GNU General Public License as published by
> +# the Free Software Foundation; either version 2 of the License, or
> +# (at your option) any later version.
> +#
> +# This program is distributed in the hope that it will be useful,
> +# but WITHOUT ANY WARRANTY; without even the implied warranty of
> +# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
> +# GNU General Public License for more details.
> +#
> +# You should have received a copy of the GNU General Public License
> +# along with this program; if not, write to the Free Software
> +# Foundation, Inc., 51 Franklin Street, Fifth Floor, Boston, MA 02110-1301 USA.
> +
> +# Embed code or other content into an OCaml file.
> +#
> +# It is embedded into a string. As OCaml string literals have virtually
> +# no restrictions on length or content we only have to escape double
> +# quotes for backslash characters.
> +
> +if [ $# -ne 3 ]; then
> + echo "embed.sh identifier input output"
> + exit 1
> +fi
> +
> +ident="$1"
> +input="$2"
> +output="$3"
> +
> +rm -f "$output" "$output"-t
> +
> +exec >"$output"-t
> +
> +echo "(* Generated by embed.sh from $input *)"
> +echo
> +echo let "$ident" = '"'
> +sed -e 's/\(["\]\)/\\\1/g' < "$input"
> +echo '"'
> +
> +chmod -w "$output"-t
> +mv "$output"-t "$output"
> diff --git a/v2v/output_rhv_upload.ml b/v2v/output_rhv_upload.ml
> new file mode 100644
> index 000000000..1be755a25
> --- /dev/null
> +++ b/v2v/output_rhv_upload.ml
> @@ -0,0 +1,390 @@
> +(* virt-v2v
> + * Copyright (C) 2009-2018 Red Hat Inc.
> + *
> + * This program is free software; you can redistribute it and/or modify
> + * it under the terms of the GNU General Public License as published by
> + * the Free Software Foundation; either version 2 of the License, or
> + * (at your option) any later version.
> + *
> + * This program is distributed in the hope that it will be useful,
> + * but WITHOUT ANY WARRANTY; without even the implied warranty of
> + * MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
> + * GNU General Public License for more details.
> + *
> + * You should have received a copy of the GNU General Public License along
> + * with this program; if not, write to the Free Software Foundation, Inc.,
> + * 51 Franklin Street, Fifth Floor, Boston, MA 02110-1301 USA.
> + *)
> +
> +open Printf
> +open Unix
> +
> +open Std_utils
> +open Tools_utils
> +open Unix_utils
> +open Common_gettext.Gettext
> +
> +open Types
> +open Utils
> +
> +type rhv_options = {
> + rhv_cafile : string;
> + rhv_cluster : string option;
> + rhv_direct : bool;
> + rhv_verifypeer : bool;
> +}
> +
> +let print_output_options () =
> + printf (f_"Output options (-oo) which can be used with -o rhv-upload:
> +
> + -oo rhv-cafile=CA.PEM Set ‘ca.pem’ certificate bundle filename.
> + -oo rhv-cluster=CLUSTERNAME Set RHV cluster name.
> + -oo rhv-direct[=true|false] Use direct transfer mode (default: false).
> + -oo rhv-verifypeer[=true|false] Verify server identity (default: false).
> +")
> +
> +let parse_output_options options =
> + let rhv_cafile = ref None in
> + let rhv_cluster = ref None in
> + let rhv_direct = ref false in
> + let rhv_verifypeer = ref false in
> +
> + List.iter (
> + function
> + | "rhv-cafile", v ->
> + if !rhv_cafile <> None then
> + error (f_"-o rhv-upload: -oo rhv-cafile set twice");
> + rhv_cafile := Some v
> + | "rhv-cluster", v ->
> + if !rhv_cluster <> None then
> + error (f_"-o rhv-upload: -oo rhv-cluster set twice");
> + rhv_cluster := Some v
> + | "rhv-direct", "" -> rhv_direct := true
> + | "rhv-direct", v -> rhv_direct := bool_of_string v
> + | "rhv-verifypeer", "" -> rhv_verifypeer := true
> + | "rhv-verifypeer", v -> rhv_verifypeer := bool_of_string v
> + | k, _ ->
> + error (f_"-o rhv-upload: unknown output option ‘-oo %s’") k
> + ) options;
> +
> + let rhv_cafile =
> + match !rhv_cafile with
> + | Some s -> s
> + | None ->
> + error (f_"-o rhv-upload: must use ‘-oo rhv-cafile’ to supply the path to the oVirt or RHV user’s ‘ca.pem’ file") in
> + let rhv_cluster = !rhv_cluster in
> + let rhv_direct = !rhv_direct in
> + let rhv_verifypeer = !rhv_verifypeer in
> +
> + { rhv_cafile; rhv_cluster; rhv_direct; rhv_verifypeer }
> +
> +let python3 = "python3" (* Defined by PEP 394 *)
> +let pidfile_timeout = 30
> +let finalization_timeout = 5*60
> +
> +class output_rhv_upload output_alloc output_conn
> + output_password output_storage
> + rhv_options =
> + (* Create a temporary directory which will be deleted on exit. *)
> + let tmpdir =
> + let base_dir = (open_guestfs ())#get_cachedir () in
> + let t = Mkdtemp.temp_dir ~base_dir "rhvupload." in
> + rmdir_on_exit t;
> + t in
> +
> + let diskid_file_of_id id = tmpdir // sprintf "diskid.%d" id in
> +
> + (* Write the Python precheck, plugin and create VM to a temporary file. *)
> + let precheck =
> + let precheck = tmpdir // "rhv-upload-precheck.py" in
> + with_open_out
> + precheck
> + (fun chan -> output_string chan Output_rhv_upload_precheck_source.code);
> + precheck in
> + let plugin =
> + let plugin = tmpdir // "rhv-upload-plugin.py" in
> + with_open_out
> + plugin
> + (fun chan -> output_string chan Output_rhv_upload_plugin_source.code);
> + plugin in
> + let createvm =
> + let createvm = tmpdir // "rhv-upload-createvm.py" in
> + with_open_out
> + createvm
> + (fun chan -> output_string chan Output_rhv_upload_createvm_source.code);
> + createvm in
> +
> + (* Is SELinux enabled and enforcing on the host? *)
> + let have_selinux =
> + 0 = Sys.command "getenforce 2>/dev/null | grep -isq Enforcing" in
> +
> + (* Check that nbdkit is available and new enough. *)
> + let error_unless_nbdkit_working () =
> + if 0 <> Sys.command "nbdkit --version >/dev/null" then
> + error (f_"nbdkit is not installed or not working. It is required to use ‘-o rhv-upload’. See \"OUTPUT TO RHV\" in the virt-v2v(1) manual.");
> +
> + (* Check it's a new enough version. The latest features we
> + * require are ‘--exit-with-parent’ and ‘--selinux-label’, both
> + * added in 1.1.14. (We use 1.1.16 as the minimum here because
> + * it also adds the selinux=yes|no flag in --dump-config).
> + *)
> + let lines = external_command "nbdkit --help" in
> + let lines = String.concat " " lines in
> + if String.find lines "exit-with-parent" == -1 ||
> + String.find lines "selinux-label" == -1 then
> + error (f_"nbdkit is not new enough, you need to upgrade to nbdkit ≥ 1.1.16")
> + in
> +
> + (* Check that the python3 plugin is installed and working
> + * and can load the plugin script.
> + *)
> + let error_unless_nbdkit_python3_working () =
> + let cmd = sprintf "nbdkit %s %s --dump-plugin >/dev/null"
> + python3 (quote plugin) in
> + if Sys.command cmd <> 0 then
> + error (f_"nbdkit Python 3 plugin is not installed or not working. It is required if you want to use ‘-o rhv-upload’.
> +
> +See also \"OUTPUT TO RHV\" in the virt-v2v(1) manual.")
> + in
> +
> + (* Check that nbdkit was compiled with SELinux support (for the
> + * --selinux-label option).
> + *)
> + let error_unless_nbdkit_compiled_with_selinux () =
> + let lines = external_command "nbdkit --dump-config" in
> + (* In nbdkit <= 1.1.15 the selinux attribute was not present
> + * at all in --dump-config output so there was no way to tell.
> + * Ignore this case because there will be an error later when
> + * we try to use the --selinux-label parameter.
> + *)
> + if List.mem "selinux=no" (List.map String.trim lines) then
> + error (f_"nbdkit was compiled without SELinux support. You will have to recompile nbdkit with libselinux-devel installed, or else set SELinux to Permissive mode while doing the conversion.")
> + in
> +
> + (* JSON parameters which are invariant between disks. *)
> + let json_params = [
> + "output_conn", JSON.String output_conn;
> + "output_password", JSON.String output_password;
> + "output_storage", JSON.String output_storage;
> + "output_sparse", JSON.Bool (match output_alloc with
> + | Sparse -> true
> + | Preallocated -> false);
> + "rhv_cafile", JSON.String rhv_options.rhv_cafile;
> + "rhv_cluster",
> + JSON.String (Option.default "Default" rhv_options.rhv_cluster);
> + "rhv_direct", JSON.Bool rhv_options.rhv_direct;
> +
> + (* The 'Insecure' flag seems to be a number with various possible
> + * meanings, however we just set it to True/False.
> + *
> + * https://github.com/oVirt/ovirt-engine-sdk/blob/19aa7070b80e60a4cfd910448287aecf9083acbe/sdk/lib/ovirtsdk4/__init__.py#L395
> + *)
> + "insecure", JSON.Bool (not rhv_options.rhv_verifypeer);
> + ] in
> +
> + (* nbdkit command line args which are invariant between disks. *)
> + let nbdkit_args =
> + let args = [
> + "nbdkit";
> +
> + "--foreground"; (* run in foreground *)
> + "--exit-with-parent"; (* exit when virt-v2v exits *)
> + "--newstyle"; (* use newstyle NBD protocol *)
> + "--exportname"; "/";
> +
> + "python3"; (* use the nbdkit Python 3 plugin *)
> + plugin; (* Python plugin script *)
> + ] in
> + let args = if verbose () then args @ ["--verbose"] else args in
> + let args =
> + (* label the socket so qemu can open it *)
> + if have_selinux then
> + args @ ["--selinux-label"; "system_u:object_r:svirt_t:s0"]
> + else args in
> + args in
> +
> +object
> + inherit output
> +
> + method precheck () =
> + error_unless_nbdkit_working ();
> + error_unless_nbdkit_python3_working ();
> + if have_selinux then
> + error_unless_nbdkit_compiled_with_selinux ()
> +
> + method as_options =
> + "-o rhv-upload" ^
> + (match output_alloc with
> + | Sparse -> "" (* default, don't need to print it *)
> + | Preallocated -> " -oa preallocated") ^
> + sprintf " -oc %s -op %s -os %s"
> + output_conn output_password output_storage
> +
> + method supported_firmware = [ TargetBIOS ]
> +
> + method prepare_targets source targets =
> + let output_name = source.s_name in
> + let json_params =
> + ("output_name", JSON.String output_name) :: json_params in
> +
> + (* Python code prechecks. These can't run in #precheck because
> + * we need to know the name of the virtual machine.
> + *)
> + let json_param_file = tmpdir // "params.json" in
> + with_open_out
> + json_param_file
> + (fun chan -> output_string chan (JSON.string_of_doc json_params));
> + if run_command [ python3; precheck; json_param_file ] <> 0 then
> + error (f_"failed server prechecks, see earlier errors");
> +
> + (* Create an nbdkit instance for each disk and set the
> + * target URI to point to the NBD socket.
> + *)
> + List.map (
> + fun t ->
> + let id = t.target_overlay.ov_source.s_disk_id in
> + let disk_name = sprintf "%s-%03d" output_name id in
> + let json_params =
> + ("disk_name", JSON.String disk_name) :: json_params in
> +
> + let disk_format =
> + match t.target_format with
> + | ("raw" | "qcow2") as fmt -> fmt
> + | _ ->
> + error (f_"rhv-upload: -of %s: Only output format ‘raw’ or ‘qcow2’ is supported. If the input is in a different format then force one of these output formats by adding either ‘-of raw’ or ‘-of qcow2’ on the command line.")
> + t.target_format in
> + let json_params =
> + ("disk_format", JSON.String disk_format) :: json_params in
> +
> + let disk_size = t.target_overlay.ov_virtual_size in
> + let json_params =
> + ("disk_size", JSON.Int64 disk_size) :: json_params in
> +
> + (* Ask the plugin to write the disk ID to a special file. *)
> + let diskid_file = diskid_file_of_id id in
> + let json_params =
> + ("diskid_file", JSON.String diskid_file) :: json_params in
> +
> + (* Write the JSON parameters to a file. *)
> + let json_param_file = tmpdir // sprintf "params%d.json" id in
> + with_open_out
> + json_param_file
> + (fun chan -> output_string chan (JSON.string_of_doc json_params));
> +
> + let sock = tmpdir // sprintf "nbdkit%d.sock" id in
> + let pidfile = tmpdir // sprintf "nbdkit%d.pid" id in
> +
> + (* Add common arguments to per-target arguments. *)
> + let args =
> + nbdkit_args @ [ "--pidfile"; pidfile;
> + "--unix"; sock;
> + sprintf "params=%s" json_param_file ] in
> +
> + (* Print the full command we are about to run when debugging. *)
> + if verbose () then (
> + eprintf "running nbdkit:\n";
> + List.iter (fun arg -> eprintf " %s" (quote arg)) args;
> + prerr_newline ()
> + );
> +
> + (* Start an nbdkit instance in the background. By using
> + * --exit-with-parent we don't have to worry about clean-up.
> + *)
> + let args = Array.of_list args in
> + let pid = fork () in
> + if pid = 0 then (
> + (* Child process (nbdkit). *)
> + execvp "nbdkit" args
> + );
> +
> + (* Wait for the pidfile to appear so we know that nbdkit
> + * is listening for requests.
> + *)
> + if not (wait_for_file pidfile pidfile_timeout) then (
> + if verbose () then
> + error (f_"nbdkit did not start up. See previous debugging messages for problems.")
> + else
> + error (f_"nbdkit did not start up. There may be errors printed by nbdkit above.
> +
> +If the messages above are not sufficient to diagnose the problem then add the ‘virt-v2v -v -x’ options and examine the debugging output carefully.")
> + );
> +
> + if have_selinux then (
> + (* Note that Unix domain sockets have both a file label and
> + * a socket/process label. Using --selinux-label above
> + * only set the socket label, but we must also set the file
> + * label.
> + *)
> + ignore (
> + run_command ["chcon"; "system_u:object_r:svirt_image_t:s0";
> + sock]
> + );
> + );
> + (* ... and the regular Unix permissions, in case qemu is
> + * running as another user.
> + *)
> + chmod sock 0o777;
> +
> + (* Tell ‘qemu-img convert’ to write to the nbd socket which is
> + * connected to nbdkit.
> + *)
> + let json_params = [
> + "file.driver", JSON.String "nbd";
> + "file.path", JSON.String sock;
> + "file.export", JSON.String "/";
> + ] in
> + let target_file =
> + TargetURI ("json:" ^ JSON.string_of_doc json_params) in
> + { t with target_file }
> + ) targets
> +
> + method create_metadata source targets _ guestcaps inspect target_firmware =
> + (* Get the UUIDs of each disk image. These files are written
> + * out by the nbdkit plugins on successful finalization of the
> + * transfer.
> + *)
> + let nr_disks = List.length targets in
> + let image_uuids =
> + List.map (
> + fun t ->
> + let id = t.target_overlay.ov_source.s_disk_id in
> + let diskid_file = diskid_file_of_id id in
> + if not (wait_for_file diskid_file finalization_timeout) then
> + error (f_"transfer of disk %d/%d failed, see earlier error messages")
> + (id+1) nr_disks;
> + let diskid = read_whole_file diskid_file in
> + diskid
> + ) targets in
> +
> + (* We don't have the storage domain UUID, but instead we write
> + * in a magic value which the Python code (which can get it)
> + * will substitute.
> + *)
> + let sd_uuid = "@SD_UUID@" in
> +
> + (* The volume and VM UUIDs are made up. *)
> + let vol_uuids = List.map (fun _ -> uuidgen ()) targets
> + and vm_uuid = uuidgen () in
> +
> + (* Create the metadata. *)
> + let ovf =
> + Create_ovf.create_ovf source targets guestcaps inspect
> + output_alloc
> + sd_uuid image_uuids vol_uuids vm_uuid
> + OVirt in
> + let ovf = DOM.doc_to_string ovf in
> +
> + let json_param_file = tmpdir // "params.json" in
> + with_open_out
> + json_param_file
> + (fun chan -> output_string chan (JSON.string_of_doc json_params));
> +
> + let ovf_file = tmpdir // "vm.ovf" in
> + with_open_out ovf_file (fun chan -> output_string chan ovf);
> + if run_command [ python3; createvm; json_param_file; ovf_file ] <> 0 then
> + error (f_"failed to create virtual machine, see earlier errors")
> +
> +end
> +
> +let output_rhv_upload = new output_rhv_upload
> +let () = Modules_list.register_output_module "rhv-upload"
> diff --git a/v2v/output_rhv_upload.mli b/v2v/output_rhv_upload.mli
> new file mode 100644
> index 000000000..f6cd69a61
> --- /dev/null
> +++ b/v2v/output_rhv_upload.mli
> @@ -0,0 +1,33 @@
> +(* virt-v2v
> + * Copyright (C) 2009-2018 Red Hat Inc.
> + *
> + * This program is free software; you can redistribute it and/or modify
> + * it under the terms of the GNU General Public License as published by
> + * the Free Software Foundation; either version 2 of the License, or
> + * (at your option) any later version.
> + *
> + * This program is distributed in the hope that it will be useful,
> + * but WITHOUT ANY WARRANTY; without even the implied warranty of
> + * MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
> + * GNU General Public License for more details.
> + *
> + * You should have received a copy of the GNU General Public License along
> + * with this program; if not, write to the Free Software Foundation, Inc.,
> + * 51 Franklin Street, Fifth Floor, Boston, MA 02110-1301 USA.
> + *)
> +
> +(** [-o rhv-upload] target. *)
> +
> +type rhv_options
> +(** Miscellaneous extra command line parameters used by rhv-upload. *)
> +
> +val print_output_options : unit -> unit
> +val parse_output_options : (string * string) list -> rhv_options
> +(** Print and parse rhv-upload -oo options. *)
> +
> +val output_rhv_upload : Types.output_allocation -> string -> string ->
> + string -> rhv_options -> Types.output
> +(** [output_rhv_upload output_alloc output_conn output_password output_storage
> + rhv_options]
> + creates and returns a new {!Types.output} object specialized for writing
> + output to oVirt or RHV directly via RHV APIs. *)
> diff --git a/v2v/output_rhv_upload_createvm_source.mli b/v2v/output_rhv_upload_createvm_source.mli
> new file mode 100644
> index 000000000..c1bafa15b
> --- /dev/null
> +++ b/v2v/output_rhv_upload_createvm_source.mli
> @@ -0,0 +1,19 @@
> +(* virt-v2v
> + * Copyright (C) 2018 Red Hat Inc.
> + *
> + * This program is free software; you can redistribute it and/or modify
> + * it under the terms of the GNU General Public License as published by
> + * the Free Software Foundation; either version 2 of the License, or
> + * (at your option) any later version.
> + *
> + * This program is distributed in the hope that it will be useful,
> + * but WITHOUT ANY WARRANTY; without even the implied warranty of
> + * MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
> + * GNU General Public License for more details.
> + *
> + * You should have received a copy of the GNU General Public License along
> + * with this program; if not, write to the Free Software Foundation, Inc.,
> + * 51 Franklin Street, Fifth Floor, Boston, MA 02110-1301 USA.
> + *)
> +
> +val code : string
> diff --git a/v2v/output_rhv_upload_plugin_source.mli b/v2v/output_rhv_upload_plugin_source.mli
> new file mode 100644
> index 000000000..c1bafa15b
> --- /dev/null
> +++ b/v2v/output_rhv_upload_plugin_source.mli
> @@ -0,0 +1,19 @@
> +(* virt-v2v
> + * Copyright (C) 2018 Red Hat Inc.
> + *
> + * This program is free software; you can redistribute it and/or modify
> + * it under the terms of the GNU General Public License as published by
> + * the Free Software Foundation; either version 2 of the License, or
> + * (at your option) any later version.
> + *
> + * This program is distributed in the hope that it will be useful,
> + * but WITHOUT ANY WARRANTY; without even the implied warranty of
> + * MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
> + * GNU General Public License for more details.
> + *
> + * You should have received a copy of the GNU General Public License along
> + * with this program; if not, write to the Free Software Foundation, Inc.,
> + * 51 Franklin Street, Fifth Floor, Boston, MA 02110-1301 USA.
> + *)
> +
> +val code : string
> diff --git a/v2v/output_rhv_upload_precheck_source.mli b/v2v/output_rhv_upload_precheck_source.mli
> new file mode 100644
> index 000000000..c1bafa15b
> --- /dev/null
> +++ b/v2v/output_rhv_upload_precheck_source.mli
> @@ -0,0 +1,19 @@
> +(* virt-v2v
> + * Copyright (C) 2018 Red Hat Inc.
> + *
> + * This program is free software; you can redistribute it and/or modify
> + * it under the terms of the GNU General Public License as published by
> + * the Free Software Foundation; either version 2 of the License, or
> + * (at your option) any later version.
> + *
> + * This program is distributed in the hope that it will be useful,
> + * but WITHOUT ANY WARRANTY; without even the implied warranty of
> + * MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
> + * GNU General Public License for more details.
> + *
> + * You should have received a copy of the GNU General Public License along
> + * with this program; if not, write to the Free Software Foundation, Inc.,
> + * 51 Franklin Street, Fifth Floor, Boston, MA 02110-1301 USA.
> + *)
> +
> +val code : string
> diff --git a/v2v/rhv-upload-createvm.py b/v2v/rhv-upload-createvm.py
> new file mode 100644
> index 000000000..a34627ec8
> --- /dev/null
> +++ b/v2v/rhv-upload-createvm.py
> @@ -0,0 +1,86 @@
> +# -*- python -*-
> +# oVirt or RHV upload create VM used by ‘virt-v2v -o rhv-upload’
> +# Copyright (C) 2018 Red Hat Inc.
> +#
> +# This program is free software; you can redistribute it and/or modify
> +# it under the terms of the GNU General Public License as published by
> +# the Free Software Foundation; either version 2 of the License, or
> +# (at your option) any later version.
> +#
> +# This program is distributed in the hope that it will be useful,
> +# but WITHOUT ANY WARRANTY; without even the implied warranty of
> +# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
> +# GNU General Public License for more details.
> +#
> +# You should have received a copy of the GNU General Public License along
> +# with this program; if not, write to the Free Software Foundation, Inc.,
> +# 51 Franklin Street, Fifth Floor, Boston, MA 02110-1301 USA.
> +
> +import json
> +import logging
> +import sys
> +import time
> +
> +from http.client import HTTPSConnection
> +from urllib.parse import urlparse
> +
> +import ovirtsdk4 as sdk
> +import ovirtsdk4.types as types
> +
> +# Parameters are passed in via a JSON doc from the OCaml code.
> +# Because this Python code ships embedded inside virt-v2v there
> +# is no formal API here.
> +params = None
> +ovf = None # OVF file
> +
> +if len(sys.argv) != 3:
> + raise RuntimeError("incorrect number of parameters")
> +
> +# Parameters are passed in via a JSON document.
> +with open(sys.argv[1], 'r') as fp:
> + params = json.load(fp)
> +
> +# What is passed in is a password file, read the actual password.
> +with open(params['output_password'], 'r') as fp:
> + output_password = fp.read()
> +output_password = output_password.rstrip()
> +
> +# Read the OVF document.
> +with open(sys.argv[2], 'r') as fp:
> + ovf = fp.read()
> +
> +# Parse out the username from the output_conn URL.
> +parsed = urlparse(params['output_conn'])
> +username = parsed.username or "admin at internal"
> +
> +# Connect to the server.
> +connection = sdk.Connection(
> + url = params['output_conn'],
> + username = username,
> + password = output_password,
> + ca_file = params['rhv_cafile'],
> + log = logging.getLogger(),
> + insecure = params['insecure'],
> +)
> +
> +system_service = connection.system_service()
> +
> +# Get the storage domain UUID and substitute it into the OVF doc.
> +sds_service = system_service.storage_domains_service()
> +sd = sds_service.list(search=("name=%s" % params['output_storage']))[0]
> +sd_uuid = sd.id
> +
> +ovf.replace("@SD_UUID@", sd_uuid)
> +
> +vms_service = system_service.vms_service()
> +vm = vms_service.add(
> + types.Vm(
> + cluster=types.Cluster(name = params['rhv_cluster']),
> + initialization=types.Initialization(
> + configuration = types.Configuration(
> + type = types.ConfigurationType.OVA,
> + data = ovf,
> + )
> + )
> + )
> +)
> diff --git a/v2v/rhv-upload-plugin.py b/v2v/rhv-upload-plugin.py
> new file mode 100644
> index 000000000..badfe63d8
> --- /dev/null
> +++ b/v2v/rhv-upload-plugin.py
> @@ -0,0 +1,438 @@
> +# -*- python -*-
> +# oVirt or RHV upload nbdkit plugin used by ‘virt-v2v -o rhv-upload’
> +# Copyright (C) 2018 Red Hat Inc.
> +#
> +# This program is free software; you can redistribute it and/or modify
> +# it under the terms of the GNU General Public License as published by
> +# the Free Software Foundation; either version 2 of the License, or
> +# (at your option) any later version.
> +#
> +# This program is distributed in the hope that it will be useful,
> +# but WITHOUT ANY WARRANTY; without even the implied warranty of
> +# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
> +# GNU General Public License for more details.
> +#
> +# You should have received a copy of the GNU General Public License along
> +# with this program; if not, write to the Free Software Foundation, Inc.,
> +# 51 Franklin Street, Fifth Floor, Boston, MA 02110-1301 USA.
> +
> +import builtins
> +import json
> +import logging
> +import ssl
> +import sys
> +import time
> +
> +from http.client import HTTPSConnection
> +from urllib.parse import urlparse
> +
> +import ovirtsdk4 as sdk
> +import ovirtsdk4.types as types
> +
> +# Timeout to wait for oVirt disks to change status, or the transfer
> +# object to finish initializing [seconds].
> +timeout = 5*60
> +
> +# Parameters are passed in via a JSON doc from the OCaml code.
> +# Because this Python code ships embedded inside virt-v2v there
> +# is no formal API here.
> +params = None
> +
> +def config(key, value):
> + global params
> +
> + if key == "params":
> + with builtins.open(value, 'r') as fp:
> + params = json.load(fp)
> + else:
> + raise RuntimeError("unknown configuration key '%s'" % key)
> +
> +def config_complete():
> + if params is None:
> + raise RuntimeError("missing configuration parameters")
> +
> +def open(readonly):
> + # Parse out the username from the output_conn URL.
> + parsed = urlparse(params['output_conn'])
> + username = parsed.username or "admin at internal"
> +
> + # Read the password from file.
> + with builtins.open(params['output_password'], 'r') as fp:
> + password = fp.read()
> + password = password.rstrip()
> +
> + # Connect to the server.
> + connection = sdk.Connection(
> + url = params['output_conn'],
> + username = username,
> + password = password,
> + ca_file = params['rhv_cafile'],
> + log = logging.getLogger(),
> + insecure = params['insecure'],
> + )
> +
> + system_service = connection.system_service()
> +
> + # Create the disk.
> + disks_service = system_service.disks_service()
> + if params['disk_format'] == "raw":
> + disk_format = types.DiskFormat.RAW
> + else:
> + disk_format = types.DiskFormat.COW
> + disk = disks_service.add(
> + disk = types.Disk(
> + name = params['disk_name'],
> + description = "Uploaded by virt-v2v",
> + format = disk_format,
> + initial_size = params['disk_size'],
> + provisioned_size = params['disk_size'],
> + # XXX Ignores params['output_sparse'].
> + # Handling this properly will be complex, see:
> + # https://www.redhat.com/archives/libguestfs/2018-March/msg00177.html
> + sparse = True,
> + storage_domains = [
> + types.StorageDomain(
> + name = params['output_storage'],
> + )
> + ],
> + )
> + )
> +
> + # Wait till the disk is up, as the transfer can't start if the
> + # disk is locked:
> + disk_service = disks_service.disk_service(disk.id)
> +
> + endt = time.time() + timeout
> + while True:
> + time.sleep(5)
> + disk = disk_service.get()
> + if disk.status == types.DiskStatus.OK:
> + break
> + if time.time() > endt:
> + raise RuntimeError("timed out waiting for disk to become unlocked")
> +
> + # Get a reference to the transfer service.
> + transfers_service = system_service.image_transfers_service()
> +
> + # Create a new image transfer.
> + transfer = transfers_service.add(
> + types.ImageTransfer(
> + image = types.Image(
> + id = disk.id
> + )
> + )
> + )
I'm pretty sure I already mentioned it before. It would help debugging
if we could dump the `disk.id` to the output here so it shows in
virt-v2v debug logs.
> + # Get a reference to the created transfer service.
> + transfer_service = transfers_service.image_transfer_service(transfer.id)
Dumping `transfer.id` could also be usefull... maybe? Don't know. If
this shows in imageio dameon logs or not...
> +
> + # After adding a new transfer for the disk, the transfer's status
> + # will be INITIALIZING. Wait until the init phase is over. The
> + # actual transfer can start when its status is "Transferring".
> + endt = time.time() + timeout
> + while True:
> + time.sleep(5)
> + transfer = transfer_service.get()
> + if transfer.phase != types.ImageTransferPhase.INITIALIZING:
> + break
> + if time.time() > endt:
> + raise RuntimeError("timed out waiting for transfer status " +
> + "!= INITIALIZING")
> +
> + # Now we have permission to start the transfer.
> + if params['rhv_direct']:
> + if transfer.transfer_url is None:
> + raise RuntimeError("direct upload to host not supported, " +
> + "requires ovirt-engine >= 4.2 and only works " +
> + "when virt-v2v is run within the oVirt/RHV " +
> + "environment, eg. on an ovirt node.")
spelling issue: oVirt instead of ovirt
Tomas
> + destination_url = urlparse(transfer.transfer_url)
> + else:
> + destination_url = urlparse(transfer.proxy_url)
> +
> + context = ssl.create_default_context()
> + context.load_verify_locations(cafile = params['rhv_cafile'])
> +
> + http = HTTPSConnection(
> + destination_url.hostname,
> + destination_url.port,
> + context = context
> + )
> +
> + # Save everything we need to make requests in the handle.
> + return {
> + 'can_flush': False,
> + 'can_trim': False,
> + 'can_zero': False,
> + 'connection': connection,
> + 'disk': disk,
> + 'disk_service': disk_service,
> + 'failed': False,
> + 'got_options': False,
> + 'highestwrite': 0,
> + 'http': http,
> + 'needs_auth': not params['rhv_direct'],
> + 'path': destination_url.path,
> + 'transfer': transfer,
> + 'transfer_service': transfer_service,
> + }
> +
> +# Can we issue zero, trim or flush requests?
> +def get_options(h):
> + if h['got_options']:
> + return
> + h['got_options'] = True
> +
> + http = h['http']
> + transfer = h['transfer']
> +
> + http.putrequest("OPTIONS", h['path'])
> + http.putheader("Authorization", transfer.signed_ticket)
> + http.endheaders()
> +
> + r = http.getresponse()
> + if r.status == 200:
> + # New imageio never needs authentication.
> + h['needs_auth'] = False
> +
> + j = json.loads(r.read())
> + h['can_zero'] = "zero" in j['features']
> + h['can_trim'] = "trim" in j['features']
> + h['can_flush'] = "flush" in j['features']
> +
> + # Old imageio servers returned either 405 Method Not Allowed or
> + # 204 No Content (with an empty body). If we see that we leave
> + # all the features as False and they will be emulated.
> + elif r.status == 405 or r.status == 204:
> + pass
> +
> + else:
> + raise RuntimeError("could not use OPTIONS request: %d: %s" %
> + (r.status, r.reason))
> +
> +def can_trim(h):
> + get_options(h)
> + return h['can_trim']
> +
> +def can_flush(h):
> + get_options(h)
> + return h['can_flush']
> +
> +def get_size(h):
> + return params['disk_size']
> +
> +# For documentation see:
> +# https://github.com/oVirt/ovirt-imageio/blob/master/docs/random-io.md
> +# For examples of working code to read/write from the server, see:
> +# https://github.com/oVirt/ovirt-imageio/blob/master/daemon/test/server_test.py
> +
> +def pread(h, count, offset):
> + http = h['http']
> + transfer = h['transfer']
> + transfer_service = h['transfer_service']
> +
> + http.putrequest("GET", h['path'])
> + # Authorization is only needed for old imageio.
> + if h['needs_auth']:
> + http.putheader("Authorization", transfer.signed_ticket)
> + http.putheader("Range", "bytes=%d-%d" % (offset, offset+count-1))
> + http.endheaders()
> +
> + r = http.getresponse()
> + # 206 = HTTP Partial Content.
> + if r.status != 206:
> + h['transfer_service'].pause()
> + h['failed'] = True
> + raise RuntimeError("could not read sector (%d, %d): %d: %s" %
> + (offset, count, r.status, r.reason))
> + return r.read()
> +
> +def pwrite(h, buf, offset):
> + http = h['http']
> + transfer = h['transfer']
> + transfer_service = h['transfer_service']
> +
> + count = len(buf)
> + h['highestwrite'] = max(h['highestwrite'], offset+count)
> +
> + http.putrequest("PUT", h['path'] + "?flush=n")
> + # Authorization is only needed for old imageio.
> + if h['needs_auth']:
> + http.putheader("Authorization", transfer.signed_ticket)
> + # The oVirt server only uses the first part of the range, and the
> + # content-length.
> + http.putheader("Content-Range", "bytes %d-%d/*" % (offset, offset+count-1))
> + http.putheader("Content-Length", str(count))
> + http.endheaders()
> + http.send(buf)
> +
> + r = http.getresponse()
> + if r.status != 200:
> + transfer_service.pause()
> + h['failed'] = True
> + raise RuntimeError("could not write sector (%d, %d): %d: %s" %
> + (offset, count, r.status, r.reason))
> +
> +def zero(h, count, offset, may_trim):
> + http = h['http']
> + transfer = h['transfer']
> + transfer_service = h['transfer_service']
> +
> + # Unlike the trim and flush calls, there is no 'can_zero' method
> + # so nbdkit could call this even if the server doesn't support
> + # zeroing. If this is the case we must emulate.
> + if not h['can_zero']:
> + emulate_zero(h, count, offset)
> + return
> +
> + # Construct the JSON request for zeroing.
> + buf = json.dumps({'op': "zero",
> + 'offset': offset,
> + 'size': count,
> + 'flush': False}).encode()
> +
> + http.putrequest("PATCH", h['path'])
> + http.putheader("Content-Type", "application/json")
> + http.putheader("Content-Length", len(buf))
> + http.endheaders()
> + http.send(buf)
> +
> + r = http.getresponse()
> + if r.status != 200:
> + transfer_service.pause()
> + h['failed'] = True
> + raise RuntimeError("could not zero sector (%d, %d): %d: %s" %
> + (offset, count, r.status, r.reason))
> +
> +def emulate_zero(h, count, offset):
> + # qemu-img convert starts by trying to zero/trim the whole device.
> + # Since we've just created a new disk it's safe to ignore these
> + # requests as long as they are smaller than the highest write seen.
> + # After that we must emulate them with writes.
> + if offset+count < h['highestwrite']:
> + http.putrequest("PUT", h['path'])
> + # Authorization is only needed for old imageio.
> + if h['needs_auth']:
> + http.putheader("Authorization", transfer.signed_ticket)
> + http.putheader("Content-Range",
> + "bytes %d-%d/*" % (offset, offset+count-1))
> + http.putheader("Content-Length", str(count))
> + http.endheaders()
> +
> + buf = bytearray(128*1024)
> + while count > len(buf):
> + http.send(buf)
> + count -= len(buf)
> + http.send(buffer(buf, 0, count))
> +
> + r = http.getresponse()
> + if r.status != 200:
> + transfer_service.pause()
> + h['failed'] = True
> + raise RuntimeError("could not write zeroes (%d, %d): %d: %s" %
> + (offset, count, r.status, r.reason))
> +
> +def trim(h, count, offset):
> + http = h['http']
> + transfer = h['transfer']
> + transfer_service = h['transfer_service']
> +
> + # Construct the JSON request for trimming.
> + buf = json.dumps({'op': "trim",
> + 'offset': offset,
> + 'size': count,
> + 'flush': False}).encode()
> +
> + http.putrequest("PATCH", h['path'])
> + http.putheader("Content-Type", "application/json")
> + http.putheader("Content-Length", len(buf))
> + http.endheaders()
> + http.send(buf)
> +
> + r = http.getresponse()
> + if r.status != 200:
> + transfer_service.pause()
> + h['failed'] = True
> + raise RuntimeError("could not trim sector (%d, %d): %d: %s" %
> + (offset, count, r.status, r.reason))
> +
> +def flush(h):
> + http = h['http']
> + transfer = h['transfer']
> + transfer_service = h['transfer_service']
> +
> + # Construct the JSON request for flushing.
> + buf = json.dumps({'op': "flush"}).encode()
> +
> + http.putrequest("PATCH", h['path'])
> + http.putheader("Content-Type", "application/json")
> + http.putheader("Content-Length", len(buf))
> + http.endheaders()
> + http.send(buf)
> +
> + r = http.getresponse()
> + if r.status != 200:
> + transfer_service.pause()
> + h['failed'] = True
> + raise RuntimeError("could not flush: %d: %s" % (r.status, r.reason))
> +
> +def delete_disk_on_failure(h):
> + disk_service = h['disk_service']
> + disk_service.remove()
> +
> +def close(h):
> + http = h['http']
> + connection = h['connection']
> +
> + # This is sometimes necessary because python doesn't set up
> + # sys.stderr to be line buffered and so debug, errors or
> + # exceptions printed previously might not be emitted before the
> + # plugin exits.
> + sys.stderr.flush()
> +
> + # If the connection failed earlier ensure we clean up the disk.
> + if h['failed']:
> + delete_disk_on_failure(h)
> + connection.close()
> + return
> +
> + try:
> + # Issue a flush request on close so that the data is written to
> + # persistent store before we create the VM.
> + if h['can_flush']:
> + flush(h)
> +
> + http.close()
> +
> + disk = h['disk']
> + transfer_service = h['transfer_service']
> +
> + transfer_service.finalize()
> +
> + # Wait until the transfer disk job is completed since
> + # only then we can be sure the disk is unlocked. As this
> + # code is not very clear, what's happening is that we are
> + # waiting for the transfer object to cease to exist, which
> + # falls through to the exception case and then we can
> + # continue.
> + endt = time.time() + timeout
> + try:
> + while True:
> + time.sleep(1)
> + tmp = transfer_service.get()
> + if time.time() > endt:
> + raise RuntimeError("timed out waiting for transfer " +
> + "to finalize")
> + except sdk.NotFoundError:
> + pass
> +
> + # Write the disk ID file. Only do this on successful completion.
> + with builtins.open(params['diskid_file'], 'w') as fp:
> + fp.write(disk.id)
> +
> + except:
> + # Otherwise on any failure we must clean up the disk.
> + delete_disk_on_failure(h)
> + raise
> +
> + connection.close()
> diff --git a/v2v/rhv-upload-precheck.py b/v2v/rhv-upload-precheck.py
> new file mode 100644
> index 000000000..2798a29dd
> --- /dev/null
> +++ b/v2v/rhv-upload-precheck.py
> @@ -0,0 +1,73 @@
> +# -*- python -*-
> +# oVirt or RHV pre-upload checks used by ‘virt-v2v -o rhv-upload’
> +# Copyright (C) 2018 Red Hat Inc.
> +#
> +# This program is free software; you can redistribute it and/or modify
> +# it under the terms of the GNU General Public License as published by
> +# the Free Software Foundation; either version 2 of the License, or
> +# (at your option) any later version.
> +#
> +# This program is distributed in the hope that it will be useful,
> +# but WITHOUT ANY WARRANTY; without even the implied warranty of
> +# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
> +# GNU General Public License for more details.
> +#
> +# You should have received a copy of the GNU General Public License along
> +# with this program; if not, write to the Free Software Foundation, Inc.,
> +# 51 Franklin Street, Fifth Floor, Boston, MA 02110-1301 USA.
> +
> +import json
> +import logging
> +import sys
> +import time
> +
> +from http.client import HTTPSConnection
> +from urllib.parse import urlparse
> +
> +import ovirtsdk4 as sdk
> +import ovirtsdk4.types as types
> +
> +# Parameters are passed in via a JSON doc from the OCaml code.
> +# Because this Python code ships embedded inside virt-v2v there
> +# is no formal API here.
> +params = None
> +
> +if len(sys.argv) != 2:
> + raise RuntimeError("incorrect number of parameters")
> +
> +# Parameters are passed in via a JSON document.
> +with open(sys.argv[1], 'r') as fp:
> + params = json.load(fp)
> +
> +# What is passed in is a password file, read the actual password.
> +with open(params['output_password'], 'r') as fp:
> + output_password = fp.read()
> +output_password = output_password.rstrip()
> +
> +# Parse out the username from the output_conn URL.
> +parsed = urlparse(params['output_conn'])
> +username = parsed.username or "admin at internal"
> +
> +# Connect to the server.
> +connection = sdk.Connection(
> + url = params['output_conn'],
> + username = username,
> + password = output_password,
> + ca_file = params['rhv_cafile'],
> + log = logging.getLogger(),
> + insecure = params['insecure'],
> +)
> +
> +system_service = connection.system_service()
> +
> +# Find if a virtual machine already exists with that name.
> +vms_service = system_service.vms_service()
> +vms = vms_service.list(
> + search = ("name=%s" % params['output_name']),
> +)
> +if len(vms) > 0:
> + vm = vms[0]
> + raise RuntimeError("VM already exists with name ‘%s’, id ‘%s’" %
> + (params['output_name'], vm.id))
> +
> +# Otherwise everything is OK, exit with no error.
> diff --git a/v2v/test-v2v-o-rhv-upload-oo-query.sh b/v2v/test-v2v-o-rhv-upload-oo-query.sh
> new file mode 100755
> index 000000000..29d69e1c1
> --- /dev/null
> +++ b/v2v/test-v2v-o-rhv-upload-oo-query.sh
> @@ -0,0 +1,38 @@
> +#!/bin/bash -
> +# libguestfs virt-v2v test script
> +# Copyright (C) 2018 Red Hat Inc.
> +#
> +# This program is free software; you can redistribute it and/or modify
> +# it under the terms of the GNU General Public License as published by
> +# the Free Software Foundation; either version 2 of the License, or
> +# (at your option) any later version.
> +#
> +# This program is distributed in the hope that it will be useful,
> +# but WITHOUT ANY WARRANTY; without even the implied warranty of
> +# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
> +# GNU General Public License for more details.
> +#
> +# You should have received a copy of the GNU General Public License
> +# along with this program; if not, write to the Free Software
> +# Foundation, Inc., 51 Franklin Street, Fifth Floor, Boston, MA 02110-1301 USA.
> +
> +# Test -oo "?" option.
> +
> +set -e
> +
> +$TEST_FUNCTIONS
> +skip_if_skipped
> +
> +export VIRT_TOOLS_DATA_DIR="$top_srcdir/test-data/fake-virt-tools"
> +export VIRTIO_WIN="$top_srcdir/test-data/fake-virtio-win"
> +
> +f=test-v2v-o-rhv-upload-oo-query.actual
> +rm -f $f
> +
> +$VG virt-v2v --debug-gc \
> + -o rhv-upload -oo "?" > $f
> +
> +grep -- "-oo rhv-cafile" $f
> +grep -- "-oo rhv-verifypeer" $f
> +
> +rm $f
> diff --git a/v2v/test-v2v-python-syntax.sh b/v2v/test-v2v-python-syntax.sh
> new file mode 100755
> index 000000000..b167f4610
> --- /dev/null
> +++ b/v2v/test-v2v-python-syntax.sh
> @@ -0,0 +1,45 @@
> +#!/bin/bash -
> +# libguestfs
> +# Copyright (C) 2018 Red Hat Inc.
> +#
> +# This program is free software; you can redistribute it and/or modify
> +# it under the terms of the GNU General Public License as published by
> +# the Free Software Foundation; either version 2 of the License, or
> +# (at your option) any later version.
> +#
> +# This program is distributed in the hope that it will be useful,
> +# but WITHOUT ANY WARRANTY; without even the implied warranty of
> +# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
> +# GNU General Public License for more details.
> +#
> +# You should have received a copy of the GNU General Public License
> +# along with this program; if not, write to the Free Software
> +# Foundation, Inc., 51 Franklin Street, Fifth Floor, Boston, MA 02110-1301 USA.
> +
> +set -e
> +
> +$TEST_FUNCTIONS
> +skip_if_skipped
> +
> +# Files to check.
> +files="rhv-upload-createvm.py rhv-upload-plugin.py rhv-upload-precheck.py"
> +
> +# Base version of Python.
> +python=python3
> +
> +# Checks the files are syntactically correct, but not very much else.
> +for f in $files; do
> + $python -m py_compile $f
> +done
> +
> +# Checks the files correspond to PEP8 coding style.
> +# https://www.python.org/dev/peps/pep-0008/
> +if $python-pep8 --version >/dev/null 2>&1; then
> + for f in $files; do
> + # Ignore:
> + # E226 missing whitespace around arithmetic operator
> + # E251 unexpected spaces around keyword / parameter equals
> + # E302 expected 2 blank lines, found 1
> + $python-pep8 --ignore=E226,E251,E302 $f
> + done
> +fi
> diff --git a/v2v/virt-v2v.pod b/v2v/virt-v2v.pod
> index 5102e6841..4eab5a55d 100644
> --- a/v2v/virt-v2v.pod
> +++ b/v2v/virt-v2v.pod
> @@ -6,15 +6,18 @@ virt-v2v - Convert a guest to use KVM
>
> virt-v2v -ic vpx://vcenter.example.com/Datacenter/esxi vmware_guest
>
> - virt-v2v -ic vpx://vcenter.example.com/Datacenter/esxi vmware_guest \
> - -o rhv -os rhv.nfs:/export_domain --bridge ovirtmgmt
> -
> virt-v2v -i libvirtxml guest-domain.xml -o local -os /var/tmp
>
> virt-v2v -i disk disk.img -o local -os /var/tmp
>
> virt-v2v -i disk disk.img -o glance
>
> + virt-v2v -ic vpx://vcenter.example.com/Datacenter/esxi vmware_guest \
> + -o rhv-upload -oc https://ovirt-engine.example.com/ovirt-engine/api \
> + -os ovirt-data -op /tmp/ovirt-admin-password -of raw \
> + -oo rhv-cafile=/tmp/ca.pem -oo rhv-direct \
> + --bridge ovirtmgmt
> +
> virt-v2v -ic qemu:///system qemu_guest --in-place
>
> =head1 DESCRIPTION
> @@ -52,20 +55,18 @@ For more information see L</INPUT FROM VMWARE VCENTER SERVER> below.
> =head2 Convert from VMware to RHV/oVirt
>
> This is the same as the previous example, except you want to send the
> -guest to a RHV-M Export Storage Domain which is located remotely
> -(over NFS) at C<rhv.nfs:/export_domain>. If you are unclear about
> -the location of the Export Storage Domain you should check the
> -settings on your RHV-M management console. Guest network
> +guest to a RHV Data Domain using the RHV REST API. Guest network
> interface(s) are connected to the target network called C<ovirtmgmt>.
>
> virt-v2v -ic vpx://vcenter.example.com/Datacenter/esxi vmware_guest \
> - -o rhv -os rhv.nfs:/export_domain --bridge ovirtmgmt
> + -o rhv-upload -oc https://ovirt-engine.example.com/ovirt-engine/api \
> + -os ovirt-data -op /tmp/ovirt-admin-password -of raw \
> + -oo rhv-cafile=/tmp/ca.pem -oo rhv-direct \
> + --bridge ovirtmgmt
>
> In this case the host running virt-v2v acts as a B<conversion server>.
>
> -Note that after conversion, the guest will appear in the RHV-M Export
> -Storage Domain, from where you will need to import it using the RHV-M
> -user interface. (See L</OUTPUT TO RHV>).
> +For more information see L</OUTPUT TO RHV> below.
>
> =head2 Convert from ESXi hypervisor over SSH to local libvirt
>
> @@ -129,9 +130,9 @@ qemu, do:
> Xen ───▶│ -i libvirt ──▶ │ │ │ (default) │
> ... ───▶│ (default) │ │ │ ──┐ └────────────┘
> └────────────┘ │ │ ─┐└──────▶ -o glance
> - -i libvirtxml ─────────▶ │ │ ┐└─────────▶ -o rhv
> - -i vmx ────────────────▶ │ │ └──────────▶ -o vdsm
> - └────────────┘
> + -i libvirtxml ─────────▶ │ │ ┐├─────────▶ -o rhv
> + -i vmx ────────────────▶ │ │ │└─────────▶ -o vdsm
> + └────────────┘ └──────────▶ -o rhv-upload
>
> Virt-v2v has a number of possible input and output modes, selected
> using the I<-i> and I<-o> options. Only one input and output mode can
> @@ -164,8 +165,9 @@ libvirt configuration file (mainly for testing).
> I<-o qemu> writes to a local disk image with a shell script for
> booting the guest directly in qemu (mainly for testing).
>
> -I<-o rhv> is used to write to a RHV / oVirt target. I<-o vdsm>
> -is only used when virt-v2v runs under VDSM control.
> +I<-o rhv-upload> is used to write to a RHV / oVirt target. I<-o rhv>
> +is a legacy method to write to RHV / oVirt E<lt> 4.2. I<-o vdsm> is
> +only used when virt-v2v runs under VDSM control.
>
> I<--in-place> instructs virt-v2v to customize the guest OS in the input
> virtual machine, instead of creating a new VM in the target hypervisor.
> @@ -550,6 +552,10 @@ written.
>
> This is the same as I<-o rhv>.
>
> +=item B<-o> B<ovirt-upload>
> +
> +This is the same as I<-o rhv-upload>.
> +
> =item B<-o> B<qemu>
>
> Set the output method to I<qemu>.
> @@ -574,6 +580,16 @@ I<-os> parameter must also be used to specify the location of the
> Export Storage Domain. Note this does not actually import the guest
> into RHV. You have to do that manually later using the UI.
>
> +See L</OUTPUT TO RHV (OLD METHOD)> below.
> +
> +=item B<-o> B<rhv-upload>
> +
> +Set the output method to I<rhv-upload>.
> +
> +The converted guest is written directly to a RHV Data Domain.
> +This is a faster method than I<-o rhv>, but requires oVirt
> +or RHV E<ge> 4.2.
> +
> See L</OUTPUT TO RHV> below.
>
> =item B<-o> B<vdsm>
> @@ -615,7 +631,33 @@ the output name is the same as the input name.
> Set output option(s) related to the current output mode.
> To display short help on what options are available you can use:
>
> - virt-v2v -o vdsm -oo "?"
> + virt-v2v -o rhv-upload -oo "?"
> +
> +=item B<-oo rhv-cafile=>F<ca.pem>
> +
> +For I<-o rhv-upload> (L<OUTPUT TO RHV>) only, the F<ca.pem> file
> +(Certificate Authority), copied from F</etc/pki/ovirt-engine/ca.pem>
> +on the oVirt engine.
> +
> +=item B<-oo rhv-cluster=>C<CLUSTERNAME>
> +
> +For I<-o rhv-upload> (L<OUTPUT TO RHV>) only, set the RHV Cluster
> +Name. If not given it uses C<Default>.
> +
> +=item B<-oo rhv-direct>
> +
> +For I<-o rhv-upload> (L<OUTPUT TO RHV>) only, if this option is given
> +then virt-v2v will attempt to directly upload the disk to the oVirt
> +node, otherwise it will proxy the upload through the oVirt engine.
> +Direct upload requires that you have network access to the oVirt
> +nodes. Non-direct upload is slightly slower but should work in all
> +situations.
> +
> +=item B<-oo rhv-verifypeer>
> +
> +For I<-o rhv-upload> (L<OUTPUT TO RHV>) only, verify the oVirt/RHV
> +server’s identity by checking the server‘s certificate against the
> +Certificate Authority.
>
> =item B<-oo vdsm-compat=0.10>
>
> @@ -1884,6 +1926,68 @@ Define the final guest in libvirt:
>
> =head1 OUTPUT TO RHV
>
> +This new method to upload guests to oVirt or RHV directly via the REST
> +API requires oVirt/RHV E<ge> 4.2.
> +
> +You need to specify I<-o rhv-upload> as well as the following extra
> +parameters:
> +
> +=over 4
> +
> +=item I<-oc> C<https://ovirt-engine.example.com/ovirt-engine/api>
> +
> +The URL of the REST API which is usually the server name with
> +C</ovirt-engine/api> appended, but might be different if you installed
> +oVirt Engine on a different path.
> +
> +You can optionally add a username and port number to the URL. If the
> +username is not specified then virt-v2v defaults to using
> +C<admin at internal> which is the typical superuser account for oVirt
> +instances.
> +
> +=item I<-of raw>
> +
> +Currently you must use I<-of raw> and you cannot use I<-oa preallocated>.
> +
> +These restrictions will be loosened in a future version.
> +
> +=item I<-op> F<password-file>
> +
> +A file containing a password to be used when connecting to the oVirt
> +engine. Note the file should contain the whole password, B<without
> +any trailing newline>, and for security the file should have mode
> +C<0600> so that others cannot read it.
> +
> +=item I<-os> C<ovirt-data>
> +
> +The storage domain.
> +
> +=item I<-oo rhv-cafile=>F<ca.pem>
> +
> +The F<ca.pem> file (Certificate Authority), copied from
> +F</etc/pki/ovirt-engine/ca.pem> on the oVirt engine.
> +
> +=item I<-oo rhv-cluster=>C<CLUSTERNAME>
> +
> +Set the RHV Cluster Name. If not given it uses C<Default>.
> +
> +=item I<-oo rhv-direct>
> +
> +If this option is given then virt-v2v will attempt to directly upload
> +the disk to the oVirt node, otherwise it will proxy the upload through
> +the oVirt engine. Direct upload requires that you have network access
> +to the oVirt nodes. Non-direct upload is slightly slower but should
> +work in all situations.
> +
> +=item I<-oo rhv-verifypeer>
> +
> +Verify the oVirt/RHV server’s identity by checking the server‘s
> +certificate against the Certificate Authority.
> +
> +=back
> +
> +=head1 OUTPUT TO RHV (OLD METHOD)
> +
> This section only applies to the I<-o rhv> output mode. If you use
> virt-v2v from the RHV-M user interface, then behind the scenes the
> import is managed by VDSM using the I<-o vdsm> output mode (which end
> --
> 2.16.2
>
--
Tomáš Golembiovský <tgolembi at redhat.com>
More information about the Libguestfs
mailing list