[Pulp-list] Yum/urlgrabber usage and multiple threads

[John Matthews]

Our usage of yum to determine the rpms in a repo has an issue.  With multiple threads calling 'pulp.server.util.get_repo_packages()' concurrently we see pycurl errors and segmentation faults.  I've added a lock which will cause these requests to be serialized.  Typically this method is only called after content has been downloaded and pulp is determining what rpms need to be imported.  

Here is the bug related to this:
BZ 695743:  https://bugzilla.redhat.com/show_bug.cgi?id=695743

These are the 2 issues I've seen:
1) With 2 threads accessing util.get_repo_packages() we'd see: A pycurl.error of: "error: cannot invoke setopt() - perform() is currently running"
2) More than 2 threads often would see a segmentation fault
  #0  file_done (conn=0x7f1650009fc0, status=CURLE_OK, premature=false) at file.c:283
  #1  0x000000353ec26b29 in Curl_done (connp=0x7f1661ff7830, status=CURLE_OK, premature=false) at url.c:5179
  #2  0x000000353ec2dda9 in Curl_do_perform (data=0x22565c0) at transfer.c:2107
  #3  0x00007f166bac5843 in do_curl_perform (self=0x2255da0) at src/pycurl.c:1024


Long term fix is to work with yum developers and break out the downloading so it can handle multiple requests at same time.