[Pulp-list] Bandwidth Limit pulp collectively over all syncs
ben.stanley at exemail.com.au
ben.stanley at exemail.com.au
Tue Jan 6 00:42:59 UTC 2015
I have researched the bandwidth limiting problem a bit, and I have come up
with a solution that works for me.
Note that I only have one pulp server, and I access the internet through a
proxy server. This simplifies my problem considerably.
I use the following iptables commands to perform rate limiting:
# Insert the numeric address of your proxy server here
PROXY_IP=XXX.XXX.XXX.XXX
# insert the port used by your proxy server here
PROXY_PORT=8080
iptables --append INPUT --protocol tcp \
--src $PROXY_IP --source-port $PROXY_PORT \
--match hashlimit \
--hashlimit-upto 200kb/sec \
--hashlimit-mode srcip \
--hashlimit-name proxy \
--jump ACCEPT
iptables --append INPUT --protocol tcp \
--src $PROXY_IP --source-port $PROXY_PORT \
--jump REJECT
I found that the actual data rate appearing on the wire was higher than
the limit, due to the reject traffic. The reject traffic was up to 50% of
the accept traffic, so this method does not appear to be very efficient.
I also found that I had to limit the number of pulp workers
(/etc/defaults/pulp_workers, PULP_CONCURRENCY=4), or the number of
downloads caused the REJECT rate to overwhelm the ACCEPT rate, and the
wire traffic becomes too high. I monitored the traffic using nload -u H,
but note that this measures all the traffic on an interface.
I found that the Firefox extension "Down Them All" implements a rate
limiting feature that does what it says. It is simpler to do in a single
process. To implement useful bandwidth limiting in pulp, there must be
communication between workers requesting a bandwidth allocation, and
relinquishing it when finished.
Ben.
More information about the Pulp-list
mailing list