[Pulp-list] pulp v1 vs pulp v2 rpm repo sync times

[Jay Dobies]

Touch base with Randy. He tweaked a bunch of those numbers before v2 
released and should be able to point you to the best places to start 
playing with.

On 02/22/2013 08:12 PM, Mike McCune wrote:
> I looked at the list and didn't see any threads around a noticeable
> difference in syncing content with pulp v1 vs v2 and wanted to start a
> general thread on the performance differences.  I searched in BZ for
> existing bugs around this subject but couldn't see anything.
>
> == Test Conditions ==
>
> hardware: 2 RHEL6.4 VMs with 1 VCPU, 2G RAM, 10G disk
>
> Just some quick tests on 2 VM's I have here at my house.  Both are
> identical, running RHEL6.4.  RPM versions:
>
> # rpm -q pulp
> pulp-1.1.15-1.el6cf.noarch
>
> # rpm -q pulp-server
> pulp-server-2.0.8-1.noarch
>
> both are 'vanilla' installs, no Katello or anything non standard.  The
> repo is public here:  http://mmccune.fedorapeople.org/repos/medium/ but
> I brought it down local to remove any network from the test to a host on
> my home network, 172.31.1.103
>
> the repo is ~150M with 2k packages.
>
> == Pulp V1 ==
>
> [root at g-02221312 ~]# pulp-admin repo create --id=test
> --feed=http://172.31.1.103/medium/
> Successfully created repository [ test ]
>
> [root at g-02221312 ~]# time pulp-admin repo sync --id=test -F
> Sync for repository test started
> Sync: Finished
> 2002/2002 new items downloaded
> 0/2002 existing items processed
>
> Item Details:
> RPMs: 2002/2002
>
>
> TIME:  1m18.686s
>
> == Pulp V2 ==
>
> # pulp-admin -u admin -p admin  rpm repo create --repo-id=test
> --feed=http://172.31.1.103/medium/
> ..
>
> # time pulp-admin -u admin -p admin  rpm repo sync run --repo-id=test
> +----------------------------------------------------------------------+
>                      Synchronizing Repository [test]
> +----------------------------------------------------------------------+
>
> This command may be exited by pressing ctrl+c without affecting the actual
> operation on the server.
> [..snip...]
> ... completed
>
> TIME:  3m12.394s
>
> So I'm hoping there is some simple tuning that can be done that can
> improve this around 3X time increase in speeds between syncing the same
> repo in V1 -> V2.  On much larger repos we have seen the sync times get
> even worse between the 2 versions.
>
> My first thought would be that settings in server.conf should get tweaked:
>
> [tasks]
> concurrency_threshold: 9
> dispatch_interval: 0.5
> archived_call_lifetime: 48
> consumer_content_weight: 0
> create_weight: 0
> publish_weight: 1
> sync_weight: 2
>
> or should I specify the --num-threads when creating the repo?
>
> Mike


-- 
Jay Dobies
Freenode: jdob @ #pulp
http://pulpproject.org