[Pulp-list] Sync all packages from PyPi with pulp_python plugin

[Randy Barlow]

-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA512

On 05/13/2015 08:21 AM, Ashby, Jason (IMS) wrote:
> I’m looking to set up a pypi mirror with pulp.  I’m currently
> using Bandersnatch for this, but it’d be nice to drop it and use
> pulp instead.  Per the docs*, I see that you can sync specific
> packages from pypi, e.g.
> 
> pulp-admin python repo create --repo-id pypi --feed 
> https://pypi.python.org/ --package-names numpy,scipy
> 
> but I can’t seem to sync ALL packages.  I tried leaving off the 
> --package-names option, but a sync downloads 0 packages.   Should
> I submit an issue/feature request at 
> https://pulp.plan.io/projects/pulp_python/issues?

Hi Jason!

The problem is that PyPI does not have one single manifest file for
the available package versions, but rather one manifest per package
name. Due to this, in order to sync all packages from PyPI it would be
necessary to make around 45-50,000 web requests just to find out what
would need to be downloaded, and then of course we would need to
perform the actual package downloads.

That said, we are working on a plan to have Pulp be able to lazy fetch
packages as they are requested. This plan will take a long time to
implement (so don't expect it in any of our close releases) but I
think it will solve this problem in a performant way.

Another possible solution may be Warehouse[0]. I've been talking to
the PyPA developers about this problem, and they are aware that it
needs to be solved. They may fix it there, in which case we can get
the Python importer to be aware of all the packages.

I have also considered just doing the 50k requests anyway. I suspect
that PyPI won't like if we do that, but it is technically possible as
well.

I say go ahead and file an RFE. I'll think some more about how we
might be able to get it working. Thanks for the note, and I hope you
enjoy the plugin otherwise!

[0] https://warehouse.python.org/

- -- 
Randy Barlow
-----BEGIN PGP SIGNATURE-----
Version: GnuPG v2

iQIcBAEBCgAGBQJVU0rsAAoJEIyFaKUJtmpiaUMP/1YfE0nNIum6EQSobZ0UPUDn
Ji9OiDLjsWDras0JEFdDOg8IS6Go2IZ3LThIOKIHXMNvMk3OAIC93dfbukRusvP3
BoyYlpZIyPS+24hbP7VPeLVcRzhC6NAf36rCdDjLtrGZGdfs2xK6h3YoPsdigqzK
9jhFFP6Det9yhqyL4IgLyCps7BrmpnQ9zHR9jbGjl0afIRhgAEjZaglBUlhXql0W
RIji9tsW3a2Cj8y/N472fY9YL9RKiossCMTw2pMY3+87zt7ihHHRuzxye4ZZixRf
PDDKfWLzDawxZyWbvKEvTdTYNeoYQ0YvHQ7D7q/x00TYm2wT/e2xsKfQcIxkzp6s
7eN3E76D7qlGrKjU2T9Jemxj6A+qidTc6AUcVyWxGb3dseRJ+vuhfzvpNuWGLSVn
N16GnEgesUn2Dl1aY763CHntlBW2mrmCrRjWAXp6s+Y62f2DNo7yN8mRcPfeSn6J
ypzo3CWv0C6PmK61RKZ7zkvTyYOi9jD+gyTAO3e9UsQNHdUDpsMW1ZGu2uaPzBLc
j578fcuspLsVYML0WKpg7wBsRSOy9O3++5XPQgPm1TUP6gm9jRaM+xNlvrZ0B0hf
mt+z9+Lf5A/I7j8/kcJ+3oTpsLovb7CQRRjagbM4W5Al3WQdVueH1gcy2YAv0zbf
XtYScIpLWhZxHHuaGgyw
=OD1J
-----END PGP SIGNATURE-----