[Pulp-list] Stress Testing Script to Create Hundreds of Repos

[Brian Bouterse]

-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA256

Hi Ben,

Regarding the tasks getting stuck, what version of Pulp are you running?

- -Brian


On 05/04/2015 02:03 AM, ben.stanley at exemail.com.au wrote:
> As I have created a script to create my repos, and it seems to
> cause pulp a lot of trouble, I thought I should share it here as a
> stress testing example.
> 
> Please find attached CreateRHELPulpRepos.sh .
> 
> I suggest you DON'T use it on an existing system, as it will likely
> stomp all over your repos.
> 
> I use it like this: 
> --------------------------------------------------------------------
>
> 
# source ./CreateRHELPulpRepos.sh
> # ConfigureRedHatWorkstationRepos 
> --------------------------------------------------------------------
>
>  Note that this script will create 278 repos.
> 
> While I run this script, I monitor: 1) The output of
> ConfigureRedHatWorkstationRepos 2) /var/log/pulp.log 3) nload (note
> that I have an incoming bandwidth limit set)
> 
> Sometimes I observe that a particular task gets stuck, and won't
> progress, despite no other tasks making progress either. Lack of
> network load is the easiest indicator to track. In this case, I
> have had to re-start pulp to get around this.
> 
> Note that the ConfigureRedHatWorkstationRepos does not actually
> sync the repos. It does create a sync schedule, but if you don't
> want to wait for the weekend, try this:
> 
> --------------------------------------------------------------------
>
> 
# source ./CreateRHELPulpRepos.sh
> # SyncAllRepos 
> --------------------------------------------------------------------
>
> 
This will sync all of the repos in pulp, one by one (provided nothing ge
ts
> stuck...).
> 
> If everything stuffs up and you want to start again, you might need
> this thermonuclear sequence: 
> --------------------------------------------------------------------
>
> 
# source ./CreateRHELPulpRepos.sh
> # while true; do sleep 10; date; CancelAllRunningTasks; done  #
> <--- CTRL-C when no tasks have been deleted for a while. #
> DeleteAllRepos 
> --------------------------------------------------------------------
>
> 
The packages remain on disk, it is just that the repos and the metadata
> are removed from mongodb. The repos can be re-created using 
> ConfigureRedHatWorkstationRepos. Only the metadata will be
> downloaded again (however, this does take time).
> 
> I hope that this script proves to be useful for stress-testing
> pulp. If this isn't enough, I can send you a few more functions to
> create several hundred more repos, although I'm sure you can see
> how to add a few more repos yourselves. Hint: s/workstation/server/
> .
> 
> Ben Stanley.
> 
> 
> 
> _______________________________________________ Pulp-list mailing
> list Pulp-list at redhat.com 
> https://www.redhat.com/mailman/listinfo/pulp-list
> 
-----BEGIN PGP SIGNATURE-----
Version: GnuPG v2

iQEcBAEBCAAGBQJVR2V5AAoJEK48cdELyEfyjtAH/3BVuFcCkQ68ohZGMyZZCyvO
UTtCCbTBsI7YZeMWlN7EeZgim7vnUSpNeBTbMf/h7JjOHC49zx5zJ/z+bZpxE3vQ
thkd3wTolw+Eg6giXJQTckZU+1mA14dTGfVnnZ4qjZLd+mWvQuq+0vRz20+FDiNa
GQeYs07w+8LxUIdnmJJLTQNX/wNihnNfyeno3vda+sZXl8L7A8CjFvKXJeN+rDBj
GvDs550eKRbzBzNyaxJSMLWp4mSUeHKN/SAqTwOli60l63XfyZq6+9R28YjHNT+u
gWbCsBn4t7YjJgy4AiTdYCCF+Wwb8JC1xLTL74hWAVQgLv0F5lbBFdO2BVjVfbw=
=SG9p
-----END PGP SIGNATURE-----