[Pulp-list] Children resource usage

Barnaby Court bcourt at redhat.com
Mon Jun 22 17:13:37 UTC 2015


Hi Sean,

Nodes is designed to help with replicating data between data centers. If you are working to scale up the load that you can handle within a single datacenter you would be better off adding some combination of more workers and  apache processes on a separate system and use NFS to share the disk. Understanding a little bit more about what bottlenecks you have encountered and what type of usage you are trying to support would help us to provide the best advice on the different ways that Pulp can be configured. Generally speaking the minimum recommended memory for a Pulp server is 4 GB of memory. Regards,

-Barnaby 

----- Original Message -----
From: "Sean Waite" <swaite at tracelink.com>
To: "Brian Bouterse" <bbouters at redhat.com>
Cc: pulp-list at redhat.com
Sent: Monday, June 22, 2015 10:11:40 AM
Subject: Re: [Pulp-list] Children resource usage

By children, I'm referring to child nodes - the subservers that can sync from a "parent" node. 

Looking again at the resources, below is what I have. It does look like the 1.7g proc is actually a worker. 

Some statistics on what I have here (resident memory): 
2 celery__main__worker procs listed as "resource_manager" - 41m memory each 
2 celery__main__worker procs listed as "reserved_resource_worker" - 42m and 1.7g respectively 
1 mongo process - 972m 
1 celerybeat - 24m 
a pile of httpd procs - 14m each 
1 qpid - 21m 

For disk utilization, the mongo db is around 3.8G and my directory containing all of the rpms etc is around 95G. 

We're on a system with only 3.5G available memory, which is probably part of the problem. We're looking at expanding it, I'm just trying to figure out how much to expand it by. From your numbers above, we'd need 6-7G of memory + 2*N gigs for the workers. Should I expect maybe 3-4 workers at any one time? I've got 2 now, but that is at an idle state. 


On Mon, Jun 22, 2015 at 9:24 AM, Brian Bouterse < bbouters at redhat.com > wrote: 


-----BEGIN PGP SIGNED MESSAGE----- 
Hash: SHA256 

Hi Sean, 

I'm not really sure what you mean by the term 'children'. Maybe you 
mean process or consumer? 

I expect pulp_resource_manager to use less than 1.7G of memory, but 
its possible. Memory analysis can be a little bit tricky so more 
details are needed about how this is being measured to be sure. 

The biggest memory process within Pulp by far is mongodb. If you can, 
ensure that at least 4G of RAM is available on that machine that you 
are running mongodb on. 

I looked into the docs and we don't talk much about the memory 
requirements. Feel free to file a bug on that if you want. Roughly I 
expect the following amounts of RAM to be available per process: 

pulp_celerybeat, 256MB - 512MB 
pulp_resource_manager, 256MB - 512MB 
pulp_workers. This process spawns N workers. Each worker could use 
256MB - 2GB depending on what its doing. 
httpd, 1GB 
mongodb, 4GB 
qpidd/rabbitMQ, ??? 

Note all the pulp_*, processes have a parent and child process, for 
the numbers above I consider each parent/child together. I usually 
show the inheritance using `sudo ps -awfux`. 

I'm interested to see what others think about these numbers too. 

- -Brian 


On 06/22/2015 08:46 AM, Sean Waite wrote: 
> Hi, 
> 
> I've got a pulp server running, and I'd like to add some children. 
> The server itself is a bit hard up on resources, so we're going to 
> rebuild with a larger one. How much resources would the children 
> use? Is it a fairly beefy process/memory hog? 
> 
> We've got a large number of repositories. pulp-resource-manager 
> seems to be using 1.7G of memory, with a .7G of mongodb. 
> 
> Any pointers on how much I might be able to expect? 
> 
> Thanks 
> 
> -- Sean Waite 
> swaite at tracelink.com <mailto: swaite at tracelink.com > Cloud Operations 
> Engineer GPG 3071E870 TraceLink, Inc. 
> 
> Be Excellent to Each Other 
> 
> 
> _______________________________________________ Pulp-list mailing 
> list Pulp-list at redhat.com 
> https://www.redhat.com/mailman/listinfo/pulp-list 
> 
-----BEGIN PGP SIGNATURE----- 
Version: GnuPG v2 

iQEcBAEBCAAGBQJViAx4AAoJEK48cdELyEfyjeUH/j06u2ERqrvTogSW+T3ZNYgI 
4xnkypN6/oIv87BhaVysif1adYI4R/egiIHlqHGxO0HBWm/AKQygMWJBvMMK3Dlg 
PtGdDdD7BBnGEwuTeFm0qJMlofk3PKmRaPRrFhwFe6DD/UaYgM7FSVsVbyn4zZpf 
HSMSk+j77FoEH8ExUX4i43UJOjkp1vfFgyynKMwxIHi6vLY0VDnmIS3iISlfroIA 
T+ZmS5t2u2NBU3dgTSHNlQsWP4BT2JH8VRWatoVoMc/vwlIJv+fzYn+tMAjNwKu+ 
Lepcowq7sXLQzmlqGgYpVMofcBy4Mv3V0z2tjZOzqySF7omIG5YK7uFDLISxu1g= 
=AhJm 
-----END PGP SIGNATURE----- 

_______________________________________________ 
Pulp-list mailing list 
Pulp-list at redhat.com 
https://www.redhat.com/mailman/listinfo/pulp-list 



-- 
Sean Waite swaite at tracelink.com 
Cloud Operations Engineer GPG 3071E870 
TraceLink, Inc. 

Be Excellent to Each Other 

_______________________________________________
Pulp-list mailing list
Pulp-list at redhat.com
https://www.redhat.com/mailman/listinfo/pulp-list




More information about the Pulp-list mailing list