[almighty] Build service and providers in Almighty - overview

[Tomas Nozicka]

Hi all,

this is a followup on Andy's initial mail[1] about this topic and
presents my view on what the Build Service (and providers) should look
like.

Feel free to challenge any design here as you see fit; we are still in
early stages and figuring it out ourself.

Andy published the generic design in his last email[1] at the highest
abstraction level and that stays the same. [2]

Let's start by defining what Build Service is, because the name might
seem a bit confusing; at least it was for me when we joined this
project. Build Service should provide user with the ability to
transform repository code (e.g. git branch) into artifacts (like docker
images) and the ability to *deploy* them to target environment (e.g.
OpenShift). Build Service should be able to cover the whole CI/*CD*
story, not just builds.

There is already a similar concept implemented in OpenShift called
*Pipelines* [3]. But, in ALM, we want to be more generic and provide
users with Build Service with plugable providers. That's why we are
creating Build SPI; to abstract the provider specifics away. Generally
speaking regular build (non-pipeline) job looks like a pipeline of 0
stages, which is a valid pipeline.

Although there will be most probably different providers in the future,
we will start by implementing "OpenShift Build Provider" (OSBP) that
will use OpenShift's pipelines.

There are 3 parts connected by Build SPI and Almighty public API:
 - ALM Core - Build Service
 - OSBP
 - ALM UI

Good thing is that after we define the interfaces (mainly Build SPI)
those can be worked on in parallel.

= ALM UI =
There is a lot of work done for visualizing pipelines in OpenShift's
Console. This is OSS and we could try to reuse those blocks. They will
most definitely need modifications because e.g. they read status
directly from OpenShift's API and that will be abstracted away by Build
SPI and ALM API in our case. Also I am not sure how well they visualize
pipelines without any stages. Also I remember Michael mentioning
Console uses Angular v1 and ALM UI v2.

UI will also need to be able to indirectly ask user for credentials to
build and target/prod environment's (more generally to 1-N
environments) using OAuth2. ALM's core and build providers will
authenticate only by token produced by this (OAuth) step. And the rest
of those tokens will be given to build provider as secrets for
authentication to other environments (e.g. in case of cross-cluster
deployments).


= ALM Build Service =
This will be the service in ALM core that will issue calls against
Build SPI provider. How it will be represented, mainly what information
it needs to hold, will be strict superset of provider configuration.


= Build Providers =
Build provider will comunicate with Build Service through Build SPI
which abstracts provider specifics away. It will have several functions
like:
 - CRUD operations
 - StartBuild
 - CancelBuild
 - GetStages
 - GetLogs
 - and many more (this will be in Build SPI spec)
which can all be fit to work with both pipelines and regular build jobs
with the assumption that regular build job is pipeline of 0 stages.


= Full picture =
 - ALM admin will add providers (register them) to ALM; or it will be
registered the way that ALM SPIs do... I haven looked into it yet:(
 - User will choose to add a Build Service to his project choosing
provider in UI
 - User will configure OAuth to his build environment [(URL)->(token)]
and optionally other OAuth resources [(URL, name)->(name, token)] which
will be passed to build provider as secrets that can be used e.g. for
cross-cluster deployments in that build
   Since this will be done through OAuth, ALM shall never see user's
credentials.
 - Build Service instance is created in ALM core
 - Build Service uses provider to create BuildConfig instance passing
it necessary configuration like: token, secrets[(name, token)],
repository reference, ...
 - followed by many Build SPI calls...

ALM core shall verify that all OAuth tokens are valid (not expired)
before calling Build SPI and refresh them with cooperation with ALM UI
if needed. (For the main token Build SPI will return unathorized, but
for the OAuth tokens in secrets the underlying technology does not
support that and build will be marked as failed otherwise. This might
be a common scenario since some tokens may last for about a day or so.)

Also in case ALM will run on the same OpenShift instance as you setup
for your builds, or there will be a dedicated OpenShift instance for
ALM build set up with ALM as OpenShift's identity provider, you can be
authenticated to your build environment using your ALM token without
the need to be asked for credentials. If allowed, this should be the
default.

= Next steps =
I will work on putting Build SPI proposal on paper and also on how we
should represent Build Service in core which is connected with it. 

There is also a userstory for next sprint on GitHub[3].

I will appreciate your feedback!

Regards,
Tomas

[1] - https://www.redhat.com/archives/almighty-public/2016-September/ms
g00150.html
[2] - https://drive.google.com/file/d/0B10zSvDl_cuwMHZtTmR1RVIteGM/view
 (RH Only, but there is a screenshot in attached in [1])
[3] - https://github.com/almighty/almighty-core/issues/352