[Libguestfs] [PATCH] v2v: -o rhv-upload: Optimize http request sending

[Nir Soffer]

On Thu, Jun 14, 2018 at 10:19 PM Richard W.M. Jones <rjones at redhat.com>
wrote:

> On Thu, Jun 14, 2018 at 09:16:01PM +0300, Nir Soffer wrote:
> > When sending request with small or no payload, it is simpler and
> > possibly more efficient to use the high level HTTPSConnection.request(),
> > instead of the lower level APIs.
> >
> > The only reason to use the lower level APIs is to avoid copying the
> > payload, or on python 2, to use a bigger buffer size when streaming a
> > file-like object.
>
> Change seems quite straightforward.  I'll have to test this out on my
> own oVirt system first before I can properly verify it.
>

I tested the same change with imageio example upload script, and
it *doubled* the throughput.

See https://gerrit.ovirt.org/#/c/92275/

Thanks, Rich.
>
> >  v2v/rhv-upload-plugin.py | 35 ++++++++++++++++-------------------
> >  1 file changed, 16 insertions(+), 19 deletions(-)
> >
> > diff --git a/v2v/rhv-upload-plugin.py b/v2v/rhv-upload-plugin.py
> > index ed99cc7a9..3fad865f6 100644
> > --- a/v2v/rhv-upload-plugin.py
> > +++ b/v2v/rhv-upload-plugin.py
> > @@ -263,12 +263,12 @@ def pread(h, count, offset):
> >      transfer = h['transfer']
> >      transfer_service = h['transfer_service']
> >
> > -    http.putrequest("GET", h['path'])
> > +    headers = {"Range", "bytes=%d-%d" % (offset, offset+count-1)}
> >      # Authorization is only needed for old imageio.
> >      if h['needs_auth']:
> > -        http.putheader("Authorization", transfer.signed_ticket)
> > -    http.putheader("Range", "bytes=%d-%d" % (offset, offset+count-1))
> > -    http.endheaders()
> > +        headers["Authorization"] = transfer.signed_ticket
> > +
> > +    http.request("GET", h['path'], headers=headers)
> >
> >      r = http.getresponse()
> >      # 206 = HTTP Partial Content.
> > @@ -319,11 +319,10 @@ def zero(h, count, offset, may_trim):
> >                        'size': count,
> >                        'flush': False}).encode()
> >
> > -    http.putrequest("PATCH", h['path'])
> > -    http.putheader("Content-Type", "application/json")
> > -    http.putheader("Content-Length", len(buf))
> > -    http.endheaders()
> > -    http.send(buf)
> > +    headers = {"Content-Type": "application/json",
> > +               "Content-Length", str(len(buf))}
> > +
> > +    http.request("PATCH", h['path'], body=buf, headers=headers)
> >
> >      r = http.getresponse()
> >      if r.status != 200:
> > @@ -368,11 +367,10 @@ def trim(h, count, offset):
> >                        'size': count,
> >                        'flush': False}).encode()
> >
> > -    http.putrequest("PATCH", h['path'])
> > -    http.putheader("Content-Type", "application/json")
> > -    http.putheader("Content-Length", len(buf))
> > -    http.endheaders()
> > -    http.send(buf)
> > +    headers = {"Content-Type": "application/json",
> > +               "Content-Length", str(len(buf))}
> > +
> > +    http.request("PATCH", h['path'], body=buf, headers=headers)
> >
> >      r = http.getresponse()
> >      if r.status != 200:
> > @@ -387,11 +385,10 @@ def flush(h):
> >      # Construct the JSON request for flushing.
> >      buf = json.dumps({'op': "flush"}).encode()
> >
> > -    http.putrequest("PATCH", h['path'])
> > -    http.putheader("Content-Type", "application/json")
> > -    http.putheader("Content-Length", len(buf))
> > -    http.endheaders()
> > -    http.send(buf)
> > +    headers = {"Content-Type": "application/json",
> > +               "Content-Length", str(len(buf))}
> > +
> > +    http.request("PATCH", h['path'], body=buf, headers=headers)
> >
> >      r = http.getresponse()
> >      if r.status != 200:
> > --
> > 2.17.1
>
> --
> Richard Jones, Virtualization Group, Red Hat
> http://people.redhat.com/~rjones
> Read my programming and virtualization blog: http://rwmj.wordpress.com
> virt-top is 'top' for virtual machines.  Tiny program with many
> powerful monitoring features, net stats, disk stats, logging, etc.
> http://people.redhat.com/~rjones/virt-top
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://listman.redhat.com/archives/libguestfs/attachments/20180615/00801b85/attachment.htm>