[almighty] Build service and providers in Almighty - overview

Wed Oct 19 16:22:50 UTC 2016

Comments inline.

On Po, 2016-10-17 at 10:38 +0200, Andrew Lee Rubinger wrote:
> Overall I think this is a very good summary and start to the
> discussion, as well as the userstory in [3].  I have added some color
> and clarifications here in the comments.
> 
> On Thu, Oct 13, 2016 at 5:26 PM, Tomas Nozicka <tnozicka at redhat.com>
> wrote:
> > Hi all,
> > 
> > this is a followup on Andy's initial mail[1] about this topic and
> > presents my view on what the Build Service (and providers) should
> > look
> > like.
> > 
> > Feel free to challenge any design here as you see fit; we are still
> > in
> > early stages and figuring it out ourself.
> > 
> > Andy published the generic design in his last email[1] at the
> > highest
> > abstraction level and that stays the same. [2]
> 
> Great; if we start to see holes in this design please raise the flag
> as soon as possible. :)  We'll make the necessary changes.
>  
> > Let's start by defining what Build Service is, because the name
> > might
> > seem a bit confusing; at least it was for me when we joined this
> > project. Build Service should provide user with the ability to
> > transform repository code (e.g. git branch) into artifacts (like
> > docker
> > images) and the ability to *deploy* them to target environment
> > (e.g.
> > OpenShift). Build Service should be able to cover the whole CI/*CD*
> > story, not just builds.
> 
> +1; "Build" is overloaded, and "CI/CD" can be misleading.  I'm open
> to changing the name of this, so long as it's clear that's the goal
> here is to take source from some repo at some revision and *do
> something* with it.  If this sounds unscoped and generic, that's
> because pipelines by their very nature are as well. :)
Agreed. Not sure about proper name for it, though. Generally this is
just scriptable source (revision) transformation. (Into artifact,
action, or both.)

>  
> > There is already a similar concept implemented in OpenShift called
> > *Pipelines* [3]. But, in ALM, we want to be more generic and
> > provide
> > users with Build Service with plugable providers. That's why we are
> > creating Build SPI; to abstract the provider specifics away.
> > Generally
> > speaking regular build (non-pipeline) job looks like a pipeline of
> > 0
> > stages, which is a valid pipeline.
> 
> Of 1 stage, Stage 0, right? :)
Actually I believe 0 stages is right. Here is my reasoning:
According to Jenkins if you do pipeline without any explicit stages and
run it (successfully), its UI tells you that there are no stages in the
pipeline == 0 stages. There are only actions not related to any stage.
Stage seems to be only explicitly marked group of actions by "stage"
instruction. Assuming you have some actions, then stage instruction
followed by some actions, you end up with 1 stage and some instructions
that do not belong to any stage.
So I would say there can be pipeline without stages == pipeline of 0
stages.

>  
> > Although there will be most probably different providers in the
> > future,
> > we will start by implementing "OpenShift Build Provider" (OSBP)
> > that
> > will use OpenShift's pipelines.
> 
> ...as fulfilled via Jenkins Pipeline.
>  
> > There are 3 parts connected by Build SPI and Almighty public API:
> >  - ALM Core - Build Service
> >  - OSBP
> >  - ALM UI 
> > 
> > Good thing is that after we define the interfaces (mainly Build
> > SPI)
> > those can be worked on in parallel.
> 
> Excellent; that's the dream here.  That we define:
> 
> 1) The Build SPI contract between Almighty and a Build Provider
> 2) The Build Domain objects and Service operations known by Almighty
> 3) How 2) plugs into the Almighty Public API for use by the UI
> 
> ...that gets groups working independently.
+1

>  
> > = ALM UI =
> > There is a lot of work done for visualizing pipelines in
> > OpenShift's
> > Console. This is OSS and we could try to reuse those blocks. They
> > will
> > most definitely need modifications because e.g. they read status
> > directly from OpenShift's API and that will be abstracted away by
> > Build
> > SPI and ALM API in our case. Also I am not sure how well they
> > visualize
> > pipelines without any stages. Also I remember Michael mentioning
> > Console uses Angular v1 and ALM UI v2.
> 
> Now that this discussion is kicked off here by Tomas, I'll raise the
> question with the OpenShift Dev List for exploring what opportunities
> we have to share or consume code, to the extent that is technically
> feasible.
+1

>  
> > UI will also need to be able to indirectly ask user for credentials
> > to
> > build and target/prod environment's (more generally to 1-N
> > environments) using OAuth2. ALM's core and build providers will
> > authenticate only by token produced by this (OAuth) step. And the
> > rest
> > of those tokens will be given to build provider as secrets for
> > authentication to other environments (e.g. in case of cross-cluster
> > deployments).
> 
> Or we'll need to have this info tied into the concept of an Almighty
> user account.  We need to flesh this out some more I believe; at what
> point do we gain auth to the OpenShift Online "Environment" for
> deployment?
We can have user configure several OAuth resources at "user level" and
have him choose on project level: which of those should be used for
"Build" (dev) environment and which others he wants to access for
cross-cluster deployment (prod).

>  
> > = ALM Build Service =
> > This will be the service in ALM core that will issue calls against
> > Build SPI provider. How it will be represented, mainly what
> > information
> > it needs to hold, will be strict superset of provider
> > configuration.
> 
> Almighty team gets to define this piece, assuming it knows nothing of
> the provider (ie. Jenkins or OpenShift) carrying out the tasks
> requested.  It's our abstraction layer.
+1

>  
> > = Build Providers =
> > Build provider will comunicate with Build Service through Build SPI
> > which abstracts provider specifics away. It will have several
> > functions
> > like:
> >  - CRUD operations
> >  - StartBuild
> >  - CancelBuild
> >  - GetStages
> >  - GetLogs
> >  - and many more (this will be in Build SPI spec)
> 
> +1; we need to account for all build artifacts including logs, test
> runs, coverage and static analysis reports, etc.
>  
> > which can all be fit to work with both pipelines and regular build
> > jobs
> > with the assumption that regular build job is pipeline of 0 stages.
> 
> 1 stage. :P
most probably 0 since there can be pipeline without any stages :P
(as explained above)

> 
> > = Full picture =
> >  - ALM admin will add providers (register them) to ALM; or it will
> > be
> > registered the way that ALM SPIs do... I haven looked into it yet:(
> 
> Aslak is on this "Service Registry" piece.
>  
> >  - User will choose to add a Build Service to his project choosing
> > provider in UI
> 
> For this first release, we can assume there's only one, the OpenShift
> one.
>  
> >  - User will configure OAuth to his build environment [(URL)-
> > >(token)]
> > and optionally other OAuth resources [(URL, name)->(name, token)]
> > which
> > will be passed to build provider as secrets that can be used e.g.
> > for
> > cross-cluster deployments in that build
> 
> This might be part of the "Environments" definition.  I'll get us an
> answer here from PM.
Also possible.

>  
> >    Since this will be done through OAuth, ALM shall never see
> > user's
> > credentials.
> >  - Build Service instance is created in ALM core
> 
> Build Service instance per...Almighty instance?  Almighty Project? 
> Almighty User?  Depends on what state the service itself is carrying,
> and what state we pass along as parameters in a request.  WDYT here?
>  
> Seems to me we could get away with a Build Service instance per
> provider, shared by the Almighty system, if requests are scoped to
> the Almighty project in question.
Yeah, we can probably have one Build Service per Almighty instance if
the state for it will be represented by project Build Config. (And
provider ID can be part of the state in BuildConfig.)

Build Provider will then be 1 per Almighty instance (or even more, in
theory) since it will be RESTful (stateless) service.

> 
> >  - Build Service uses provider to create BuildConfig instance
> > passing
> > it necessary configuration like: token, secrets[(name, token)],
> > repository reference, ...
> >  - followed by many Build SPI calls...
> > 
> > ALM core shall verify that all OAuth tokens are valid (not expired)
> > before calling Build SPI and refresh them with cooperation with ALM
> > UI
> > if needed. (For the main token Build SPI will return unathorized,
> > but
> > for the OAuth tokens in secrets the underlying technology does not
> > support that and build will be marked as failed otherwise. This
> > might
> > be a common scenario since some tokens may last for about a day or
> > so.)
> > 
> > Also in case ALM will run on the same OpenShift instance as you
> > setup
> > for your builds, or there will be a dedicated OpenShift instance
> > for
> > ALM build set up with ALM as OpenShift's identity provider, you can
> > be
> > authenticated to your build environment using your ALM token
> > without
> > the need to be asked for credentials. If allowed, this should be
> > the
> > default.
> 
> We can safely assume for now (through Spring) that builds will be
> executing on the same OpenShift instance that is hosting Almighty.
For OSBP it does make any difference. Maybe Build Service and UI won't
actually have to ask user to set up OAuth and just generate it
internally to the same OpenShift instance for now.

>  
> > = Next steps =
> > I will work on putting Build SPI proposal on paper and also on how
> > we
> > should represent Build Service in core which is connected with it. 
> > 
> > There is also a userstory for next sprint on GitHub[3].
> > 
> > I will appreciate your feedback!
> 
> S,
> ALR 
> > 
> > Regards,
> > Tomas
> > 
> > [1] - https://www.redhat.com/archives/almighty-public/2016-Septembe
> > r/ms
> > g00150.html
> > [2] - https://drive.google.com/file/d/0B10zSvDl_cuwMHZtTmR1RVIteGM/
> > view
> >  (RH Only, but there is a screenshot in attached in [1])
> > [3] - https://github.com/almighty/almighty-core/issues/352
> > 
> > _______________________________________________
> > almighty-public mailing list
> > almighty-public at redhat.com
> > https://www.redhat.com/mailman/listinfo/almighty-public
> > 
> 
> 
> 
> -- 
> Red Hat Developer Programs Architecture
> @ALRubinger