[Libguestfs] [PATCH] Improve memory management of nbdkit python plugin example

[Pino Toscano]

On Monday, 26 September 2016 17:30:29 CEST Carl-Daniel Hailfinger wrote:
> Hi,
> 
> sorry for the resend. My mailer mangled the previous attempt.
> 
> On 26.09.2016 17:19, Pino Toscano wrote:
> > On Monday, 26 September 2016 17:07:41 CEST Carl-Daniel Hailfinger wrote:
> >> the nbdkit python plugin example has suboptimal memory management:
> >> - it creates the disk image as a string on init
> >> - it casts the string to bytearray on every read
> >> - it copies the string before and the string after the written region,
> >> then reassembles those pieces together with the written region to a new
> >> disk image string
> >>
> >> This is not a problem as long as the image is small, but in my tests
> >> with a 5 GB sized image nbdkit already used 15 GB RAM directly after
> >> startup, and even more (20-25 GB) on the first write.
> >>
> >> This changes the code to use bytearray everywhere and use the proper
> >> methods to change bytearray objects directly. With the patch applied,
> >> nbdkit with a 5 GB image will still only use 5 GB RAM even during heavy
> >> read/write activity.
> >>
> >> Regards,
> >> Carl-Daniel
> >>
> >> diff -r 521366d1854b -r d7d5078d08c7 plugins/python/example.py
> >> --- a/plugins/python/example.py	Sun Jul 10 17:10:30 2016 +0100
> >> +++ b/plugins/python/example.py	Sun Sep 25 05:04:02 2016 +0200
> >> @@ -29,7 +29,7 @@
> >>  # reconnect to the same server you should see the same disk.  You
> >>  # could also put this into the handle, so there would be a fresh disk
> >>  # per handle.
> >> -disk = "\0" * (1024*1024);
> >> +disk = bytearray(1024 * 1024)
> >>  
> >>  # This just prints the extra command line parameters, but real plugins
> >>  # should parse them and reject any unknown parameters.
> >> @@ -50,9 +50,9 @@
> >>  
> >>  def pread(h, count, offset):
> >>      global disk
> >> -    return bytearray (disk[offset:offset+count])
> >> +    return disk[offset:offset+count]
> >>  
> >>  def pwrite(h, buf, offset):
> >>      global disk
> >>      end = offset + len (buf)
> >> -    disk = disk[:offset] + buf + disk[end:]
> >> +    disk[offset:end] = buf
> > I'd look fine to me -- I guess it should work with both Python 2 and 3,
> > right?
> 
> Indeed. By the way, does anyone know if there is a Python 3 plugin
> infrastructure for nbdkit?

nbdkit builds using the Python found at configure time.  It should be
enough to run configure with PYTHON=python3 to let it build the plugin
with Python 3; if it does not, it can be always fixed.
OTOH, even if you build twice nbdkit (one for Python 2, and the other
for Python 3), the name of the plugin does not change, so they are not
coinstallable.

-- 
Pino Toscano
-------------- next part --------------
A non-text attachment was scrubbed...
Name: signature.asc
Type: application/pgp-signature
Size: 819 bytes
Desc: This is a digitally signed message part.
URL: <http://listman.redhat.com/archives/libguestfs/attachments/20160926/0d885c3a/attachment.sig>