[Tendrl-devel] simplification of python packaging

[Martin Bukatovic]

# Tendrl Dependencies issue

Back in January, I created [issue
#18](https://github.com/Tendrl/alerting/issues/18) about specifying
dependencies in requirements.txt file, which I didn't like. Rohan argued
that Tendrl developers maintain of dependencies via `pip freeze` this
way. I accepted it and moved on.

Then during/after fixing [tox/travis ci setup
#25](https://github.com/Tendrl/alerting/pull/25) (which has been just
merged), I realized the bigger picture here and noticed few problems as
I wanted to fix remaining issues with tendrl-alerting before I create
similar pull requests for the rest of tendrl python components, as we
agreed on with Rohan and Nishanth.

1) First issue I noticed is that we have incorrect dependencies listed
in requirements.txt file in most Tendrl repositories:

* unnecessary modules are specified (not all items in tendrl-alerting
requirements.txt must be there)
* some required modules are missing (PyYAML is missing in
tendrl-alerting requirements.txt)

Those issues are then carried on into list of dependencies in rpm spec
file, making the proble worse (eg.
https://github.com/Tendrl/alerting/pull/28#discussion_r96627652).

To solve this, I would suggest to not use `pip freeze` to generate
requirements.txt file blindly (or to reuse requirements.txt from other
tendrl repo), but rather add line with name of required module along new
code which uses it. As long as one writes at least some unit tests for
the new code, one would notice missing requirement early.

So this is (imho) easy to solve and prevent.

2) Then the other issue is related to listing other tendrl component as
a requirement. For example: tendrl-alerting requires tendrl-commons to run.

Note that in this case, we:

* don't have tendrl-commons on PyPI
* we need to reffer to devel version (latest master from github) of
  tendrl-commons to run unit tests (even if we published tendrl-commons
on PyPI)

I was thinking how to solve this in a simple way and come up with the
following:

My Proposal
-----------

Some tendrl python components are actually libraries (eg.
tendrl-commons), while others are rather applications (eg.
tendrl-gluster-integration or tendrl-alerting).

Tendrl modules for libraries (such as tendrl-common) would follow
somewhat higher standars for packaging:

* installable via `pip install` (from end user perspective)
* can be published on PyPI (I'm just saying that it would be possible)
* all requirements listed directly in install_requires in setup.py
* no requirements.txt file

Tendrl applications, on the other hand:

* have all requirements listed in requirements.txt file (including link
to other tendrl libraries, via git+https url)
* don't use install_requires in setup.py file at all
* can't be installed via pip install (from end user perspective, dev
could do whathewer coding/testing requires)
* would not be published on PyPI

Why I propose drop user-end installation via pip install for Tendrl
application modules? Because full installation requires additional steps
which are out of scope of pip install anyway, such as installation of
default config files into /etc, installation of systemd unit files.
Neither can pip list system (non python requirements). Droping this use
case would simplify our packaging guidelines.

Why I don't like to read requirements.txt file and push it into
install_requires in setup.py? Because it's not necessary (see previous
paragraph) and it's actually not possible: when I specify full git+https
url of some other tendrl component like this:

git+https://github.com/Tendrl/commons.git

It's totally fine for requirements.txt file, but setup.py can't handle
it. This way, full list of requirements would be on a single place.

Why I suggest to drop requirements.txt file from tendrl libraries?
Because this file is not meant to be used for specifying dependencies of
libraries, it's not necessary.

Does that make sense for you?

What would that mean for tendrl-alerting?

* https://github.com/Tendrl/alerting/issues/17 would be closed as not
valid (that would be easy :-)
* https://github.com/Tendrl/alerting/issues/18 would remain closed
(again, quite easy)
* requirements.txt file would need to be inspected: tendrl-common github
url added, unnecessary modules removed, missing modules added
* setup.py would not contain any code which pushes requirements.txt file
content into install_requires (that is easy as well, there is no such
code yet)

-- 
Martin Bukatovic
USM QE team