package upload/import speed

[Mike Bonnet]

On Wed, 2008-07-23 at 14:03 -0700, Toshio Kuratomi wrote:
> Mike Bonnet wrote:
> > On Wed, 2008-07-23 at 15:56 -0400, Naveen Gavini wrote:
> >> Hello all,
> >>
> >> When importing or building RPM's from source is there an option that is 
> >> throttling the upload speed of the package?
> >> We are seeing speeds of 10kbps which makes importing/building large 
> >> packages very slow.
> > 
> > Package upload happens via xml-rpc, and to avoid holding the entire
> > contents of the file in memory, it's done in chunks.  Unfortunately
> > python xml-rpc libs don't support keep-alive, so if you're using SSL
> > authentication you incur the non-trivial SSL connection setup/teardown
> > overhead for every chunk, in addition to the TCP connection
> > setup/teardown.  The chunk size in your version is probably very small
> > (64k?).  That has been increased in the Koji repo, but not released yet.
> > The next version will improve this situation somewhat.
> > 
> Any chance we could port to pycurl which does support keep-alive?  Or is 
> urllib pretty deeply embedded in the code ATM?

I don't know anything about pycurl, but the transport layer is decently
separated from the XML-RPC layer in xmlrpclib, so it should be possible
to replace that it with a transport that supports SSL and keep-alive.  I
haven't looked at how much work this might be though.

> -Toshio
> 
> --
> Fedora-buildsys-list mailing list
> Fedora-buildsys-list at redhat.com
> https://www.redhat.com/mailman/listinfo/fedora-buildsys-list