[almighty] Build service and providers in Almighty - overview

[Andrew Lee Rubinger]

Overall I think this is a very good summary and start to the discussion, as
well as the userstory in [3].  I have added some color and clarifications
here in the comments.

On Thu, Oct 13, 2016 at 5:26 PM, Tomas Nozicka <tnozicka at redhat.com> wrote:

> Hi all,
>
> this is a followup on Andy's initial mail[1] about this topic and
> presents my view on what the Build Service (and providers) should look
> like.
>
> Feel free to challenge any design here as you see fit; we are still in
> early stages and figuring it out ourself.
>
> Andy published the generic design in his last email[1] at the highest
> abstraction level and that stays the same. [2]
>

Great; if we start to see holes in this design please raise the flag as
soon as possible. :)  We'll make the necessary changes.


> Let's start by defining what Build Service is, because the name might
> seem a bit confusing; at least it was for me when we joined this
> project. Build Service should provide user with the ability to
> transform repository code (e.g. git branch) into artifacts (like docker
> images) and the ability to *deploy* them to target environment (e.g.
> OpenShift). Build Service should be able to cover the whole CI/*CD*
> story, not just builds.
>

+1; "Build" is overloaded, and "CI/CD" can be misleading.  I'm open to
changing the name of this, so long as it's clear that's the goal here is to
take source from some repo at some revision and *do something* with it.  If
this sounds unscoped and generic, that's because pipelines by their very
nature are as well. :)


> There is already a similar concept implemented in OpenShift called
> *Pipelines* [3]. But, in ALM, we want to be more generic and provide
> users with Build Service with plugable providers. That's why we are
> creating Build SPI; to abstract the provider specifics away. Generally
> speaking regular build (non-pipeline) job looks like a pipeline of 0
> stages, which is a valid pipeline.
>

Of 1 stage, Stage 0, right? :)


> Although there will be most probably different providers in the future,
> we will start by implementing "OpenShift Build Provider" (OSBP) that
> will use OpenShift's pipelines.
>

...as fulfilled via Jenkins Pipeline.


> There are 3 parts connected by Build SPI and Almighty public API:
>  - ALM Core - Build Service
>  - OSBP
>  - ALM UI


> Good thing is that after we define the interfaces (mainly Build SPI)
> those can be worked on in parallel.
>

Excellent; that's the dream here.  That we define:

1) The Build SPI contract between Almighty and a Build Provider
2) The Build Domain objects and Service operations known by Almighty
3) How 2) plugs into the Almighty Public API for use by the UI

...that gets groups working independently.


> = ALM UI =
> There is a lot of work done for visualizing pipelines in OpenShift's
> Console. This is OSS and we could try to reuse those blocks. They will
> most definitely need modifications because e.g. they read status
> directly from OpenShift's API and that will be abstracted away by Build
> SPI and ALM API in our case. Also I am not sure how well they visualize
> pipelines without any stages. Also I remember Michael mentioning
> Console uses Angular v1 and ALM UI v2.
>

Now that this discussion is kicked off here by Tomas, I'll raise the
question with the OpenShift Dev List for exploring what opportunities we
have to share or consume code, to the extent that is technically feasible.


> UI will also need to be able to indirectly ask user for credentials to
> build and target/prod environment's (more generally to 1-N
> environments) using OAuth2. ALM's core and build providers will
> authenticate only by token produced by this (OAuth) step. And the rest
> of those tokens will be given to build provider as secrets for
> authentication to other environments (e.g. in case of cross-cluster
> deployments).
>

Or we'll need to have this info tied into the concept of an Almighty user
account.  We need to flesh this out some more I believe; at what point do
we gain auth to the OpenShift Online "Environment" for deployment?


> = ALM Build Service =
> This will be the service in ALM core that will issue calls against
> Build SPI provider. How it will be represented, mainly what information
> it needs to hold, will be strict superset of provider configuration.
>

Almighty team gets to define this piece, assuming it knows nothing of the
provider (ie. Jenkins or OpenShift) carrying out the tasks requested.  It's
our abstraction layer.


> = Build Providers =
> Build provider will comunicate with Build Service through Build SPI
> which abstracts provider specifics away. It will have several functions
> like:
>  - CRUD operations
>  - StartBuild
>  - CancelBuild
>  - GetStages
>  - GetLogs
>  - and many more (this will be in Build SPI spec)
>

+1; we need to account for all build artifacts including logs, test runs,
coverage and static analysis reports, etc.


> which can all be fit to work with both pipelines and regular build jobs
> with the assumption that regular build job is pipeline of 0 stages.
>

1 stage. :P

= Full picture =
>  - ALM admin will add providers (register them) to ALM; or it will be
> registered the way that ALM SPIs do... I haven looked into it yet:(
>

Aslak is on this "Service Registry" piece.


>  - User will choose to add a Build Service to his project choosing
> provider in UI
>

For this first release, we can assume there's only one, the OpenShift one.


>  - User will configure OAuth to his build environment [(URL)->(token)]
> and optionally other OAuth resources [(URL, name)->(name, token)] which
> will be passed to build provider as secrets that can be used e.g. for
> cross-cluster deployments in that build
>

This might be part of the "Environments" definition.  I'll get us an answer
here from PM.


>    Since this will be done through OAuth, ALM shall never see user's
> credentials.
>  - Build Service instance is created in ALM core
>

Build Service instance per...Almighty instance?  Almighty Project?
Almighty User?  Depends on what state the service itself is carrying, and
what state we pass along as parameters in a request.  WDYT here?

Seems to me we could get away with a Build Service instance per provider,
shared by the Almighty system, if requests are scoped to the Almighty
project in question.

 - Build Service uses provider to create BuildConfig instance passing
> it necessary configuration like: token, secrets[(name, token)],
> repository reference, ...
>  - followed by many Build SPI calls...
>
> ALM core shall verify that all OAuth tokens are valid (not expired)
> before calling Build SPI and refresh them with cooperation with ALM UI
> if needed. (For the main token Build SPI will return unathorized, but
> for the OAuth tokens in secrets the underlying technology does not
> support that and build will be marked as failed otherwise. This might
> be a common scenario since some tokens may last for about a day or so.)
>
> Also in case ALM will run on the same OpenShift instance as you setup
> for your builds, or there will be a dedicated OpenShift instance for
> ALM build set up with ALM as OpenShift's identity provider, you can be
> authenticated to your build environment using your ALM token without
> the need to be asked for credentials. If allowed, this should be the
> default.
>

We can safely assume for now (through Spring) that builds will be executing
on the same OpenShift instance that is hosting Almighty.


> = Next steps =
> I will work on putting Build SPI proposal on paper and also on how we
> should represent Build Service in core which is connected with it.
>
> There is also a userstory for next sprint on GitHub[3].
>
> I will appreciate your feedback!
>

S,
ALR

>
> Regards,
> Tomas
>
> [1] - https://www.redhat.com/archives/almighty-public/2016-September/ms
> g00150.html
> [2] - https://drive.google.com/file/d/0B10zSvDl_cuwMHZtTmR1RVIteGM/view
>  (RH Only, but there is a screenshot in attached in [1])
> [3] - https://github.com/almighty/almighty-core/issues/352
>
> _______________________________________________
> almighty-public mailing list
> almighty-public at redhat.com
> https://www.redhat.com/mailman/listinfo/almighty-public
>



-- 
Red Hat Developer Programs Architecture
@ALRubinger
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://listman.redhat.com/archives/almighty-public/attachments/20161017/be00b94d/attachment.htm>