[katello-devel] Katello upgrade redesign

[Lukas Zapletal]

Hello,

working on https://bugzilla.redhat.com/show_bug.cgi?id=868916 , I found we
have some bugs and also design issues with katello upgrade.

The current design is simple, we have two kinds of upgrade scripts which
are in two directories.

/usr/share/katello/install/upgrade-scripts/default/*

and

/usr/share/katello/install/upgrade-scripts/upgrade/*

Katello-upgrade first shuts downs all the services and then starts
upgrading by calling upgrade scripts in this order: default, upgrade.
Default scripts are executed every katello-upgrade call, while upgrade
scripts are only executed once (and then katello-upgrade "marks" them in
the /var/lib/katello/upgrade-history file (just a list of files which
have been already executed).

Default scripts (executed every time) are:

0002_update_pulp.sh
 - starts mongod
 - does pulp-migrate
0003_update_candlepin.sh
 - executes dpdb
0004_migrate_katello_db.s
 - starts candlepin and elasticsearch
 - runs katello migration

Upgrade scripts (executed only once) are:

(none for CFSE 1.0/1.1 - just two examples that does nothing)

0001_preallocate_mongo.sh
 - prepares mongo journal for new version of mongo
0002_create_foreman_user
 - creates foreman user

Now there are couple of issues which are blockers for CFSE 1.1 and
couple of design issues I would like to also fix for Katello.

Basically katello-upgrade currently does not work in CFSE 1.1 because we
have some code logic in the migration scripts (filters) which require
all the backend engines to be running. We are not starting pulp and also
elasticsearch starts slowly and we need to wait until it is available.

Also there are couple of example migration scripts which does nothing,
but creating pyc/pyo bytecode files which are then added to the executed
list. I will remove them and make sure python scripts does not create
those.

That is necessary for CFSE 1.1 upgrades, now for Katello we have a
design issue. Some scripts need the services to be running (like
pulp migration), others must not have services running (candlepin
migration).

Therefore for Katello, I plan to split our upgrade scripts into two
categories (by script numbers):

0001-2999: Will be executed when backend engines are down
3000-5999: Will be executed when backend engines are up
6000-9999: Will be executed when backend engines and Katello are up

This will be the same for both default and upgrade scripts. It would
look (default scripts) like:

0001_stop_all_services
0010_migrate_candlepin
3000_start_backend_engines
3010_migrate_pulp
3020_migrate_katello
6000_start_katello

Please note I am not increasing numbers by one, but by ten. REMEMBER
BASIC? Then you know the reason for that (we will be adding foreman
there soon).

This way we will be able to put a script (either defaul or upgrade - 
one execution only) into the desired phase easily.

In short, this is the todo:

CFSE 1.1 issues (blockers):

- services should be ALL started before katello db migration
- elasticsearch start is not blocking (start is slow and we need to wait)
- python example scripts are creating pyc/pyo objects in the upgrade folder

Katello issues (non-blockers but MUSTFIX for the next release):

- split the migration (both default and upgrade) into phases
- no way of executing custom scripts before and after services are up
- 0002_create_foreman_users script is missing header
- make sure python scripts are not creating pyc/pyo files
- numbering for default scrits should be loose (0001-0004 is too tight)
- assumeyes option does not work
- help page and man page are not complete
- write README file with tutorial how to create migration script

I am working on the fix for that bug (first part) and also adding the latter
on the backlog.

Discuss

-- 
Later,

 Lukas "lzap" Zapletal
 #katello #systemengine