[almighty] Build service and providers in Almighty - overview

Mon Oct 17 13:08:36 UTC 2016

On 17 October 2016 at 04:38, Andrew Lee Rubinger <alr at redhat.com> wrote:
> Overall I think this is a very good summary and start to the discussion, as
> well as the userstory in [3].  I have added some color and clarifications
> here in the comments.
>
> On Thu, Oct 13, 2016 at 5:26 PM, Tomas Nozicka <tnozicka at redhat.com> wrote:
>>
>> Hi all,
>>
>> this is a followup on Andy's initial mail[1] about this topic and
>> presents my view on what the Build Service (and providers) should look
>> like.
>>
>> Feel free to challenge any design here as you see fit; we are still in
>> early stages and figuring it out ourself.
>>
>> Andy published the generic design in his last email[1] at the highest
>> abstraction level and that stays the same. [2]
>
>
> Great; if we start to see holes in this design please raise the flag as soon
> as possible. :)  We'll make the necessary changes.
>
>>
>> Let's start by defining what Build Service is, because the name might
>> seem a bit confusing; at least it was for me when we joined this
>> project. Build Service should provide user with the ability to
>> transform repository code (e.g. git branch) into artifacts (like docker
>> images) and the ability to *deploy* them to target environment (e.g.
>> OpenShift). Build Service should be able to cover the whole CI/*CD*
>> story, not just builds.
>
>
> +1; "Build" is overloaded, and "CI/CD" can be misleading.  I'm open to
> changing the name of this, so long as it's clear that's the goal here is to
> take source from some repo at some revision and *do something* with it.  If
> this sounds unscoped and generic, that's because pipelines by their very
> nature are as well. :)
>
>>
>> There is already a similar concept implemented in OpenShift called
>> *Pipelines* [3]. But, in ALM, we want to be more generic and provide
>> users with Build Service with plugable providers. That's why we are
>> creating Build SPI; to abstract the provider specifics away. Generally
>> speaking regular build (non-pipeline) job looks like a pipeline of 0
>> stages, which is a valid pipeline.
>
>
> Of 1 stage, Stage 0, right? :)
>
>>
>> Although there will be most probably different providers in the future,
>> we will start by implementing "OpenShift Build Provider" (OSBP) that
>> will use OpenShift's pipelines.
>
>
> ...as fulfilled via Jenkins Pipeline.
>
>>
>> There are 3 parts connected by Build SPI and Almighty public API:
>>  - ALM Core - Build Service
>>  - OSBP
>>  - ALM UI
>>
>>
>> Good thing is that after we define the interfaces (mainly Build SPI)
>> those can be worked on in parallel.
>
>
> Excellent; that's the dream here.  That we define:
>
> 1) The Build SPI contract between Almighty and a Build Provider
> 2) The Build Domain objects and Service operations known by Almighty
> 3) How 2) plugs into the Almighty Public API for use by the UI
>
> ...that gets groups working independently.
>
>>
>> = ALM UI =
>> There is a lot of work done for visualizing pipelines in OpenShift's
>> Console. This is OSS and we could try to reuse those blocks. They will
>> most definitely need modifications because e.g. they read status
>> directly from OpenShift's API and that will be abstracted away by Build
>> SPI and ALM API in our case. Also I am not sure how well they visualize
>> pipelines without any stages. Also I remember Michael mentioning
>> Console uses Angular v1 and ALM UI v2.
>
>
> Now that this discussion is kicked off here by Tomas, I'll raise the
> question with the OpenShift Dev List for exploring what opportunities we
> have to share or consume code, to the extent that is technically feasible.

I'll also add that Todd and I reviewed those designs with Monica and
Allie on Friday and there may be a few updates proposed at the
wireframe (i.e. what's displayed) level.

>
>>
>> UI will also need to be able to indirectly ask user for credentials to
>> build and target/prod environment's (more generally to 1-N
>> environments) using OAuth2. ALM's core and build providers will
>> authenticate only by token produced by this (OAuth) step. And the rest
>> of those tokens will be given to build provider as secrets for
>> authentication to other environments (e.g. in case of cross-cluster
>> deployments).
>
>
> Or we'll need to have this info tied into the concept of an Almighty user
> account.  We need to flesh this out some more I believe; at what point do we
> gain auth to the OpenShift Online "Environment" for deployment?
>
>>
>> = ALM Build Service =
>> This will be the service in ALM core that will issue calls against
>> Build SPI provider. How it will be represented, mainly what information
>> it needs to hold, will be strict superset of provider configuration.
>
>
> Almighty team gets to define this piece, assuming it knows nothing of the
> provider (ie. Jenkins or OpenShift) carrying out the tasks requested.  It's
> our abstraction layer.
>
>>
>> = Build Providers =
>> Build provider will comunicate with Build Service through Build SPI
>> which abstracts provider specifics away. It will have several functions
>> like:
>>  - CRUD operations
>>  - StartBuild
>>  - CancelBuild
>>  - GetStages
>>  - GetLogs
>>  - and many more (this will be in Build SPI spec)
>
>
> +1; we need to account for all build artifacts including logs, test runs,
> coverage and static analysis reports, etc.
>
>>
>> which can all be fit to work with both pipelines and regular build jobs
>> with the assumption that regular build job is pipeline of 0 stages.
>
>
> 1 stage. :P
>
>> = Full picture =
>>  - ALM admin will add providers (register them) to ALM; or it will be
>> registered the way that ALM SPIs do... I haven looked into it yet:(
>
>
> Aslak is on this "Service Registry" piece.
>
>>
>>  - User will choose to add a Build Service to his project choosing
>> provider in UI
>
>
> For this first release, we can assume there's only one, the OpenShift one.
>
>>
>>  - User will configure OAuth to his build environment [(URL)->(token)]
>> and optionally other OAuth resources [(URL, name)->(name, token)] which
>> will be passed to build provider as secrets that can be used e.g. for
>> cross-cluster deployments in that build
>
>
> This might be part of the "Environments" definition.  I'll get us an answer
> here from PM.
>
>>
>>    Since this will be done through OAuth, ALM shall never see user's
>> credentials.
>>  - Build Service instance is created in ALM core
>
>
> Build Service instance per...Almighty instance?  Almighty Project?  Almighty
> User?  Depends on what state the service itself is carrying, and what state
> we pass along as parameters in a request.  WDYT here?
>
> Seems to me we could get away with a Build Service instance per provider,
> shared by the Almighty system, if requests are scoped to the Almighty
> project in question.
>
>>  - Build Service uses provider to create BuildConfig instance passing
>> it necessary configuration like: token, secrets[(name, token)],
>> repository reference, ...
>>  - followed by many Build SPI calls...
>>
>> ALM core shall verify that all OAuth tokens are valid (not expired)
>> before calling Build SPI and refresh them with cooperation with ALM UI
>> if needed. (For the main token Build SPI will return unathorized, but
>> for the OAuth tokens in secrets the underlying technology does not
>> support that and build will be marked as failed otherwise. This might
>> be a common scenario since some tokens may last for about a day or so.)
>>
>> Also in case ALM will run on the same OpenShift instance as you setup
>> for your builds, or there will be a dedicated OpenShift instance for
>> ALM build set up with ALM as OpenShift's identity provider, you can be
>> authenticated to your build environment using your ALM token without
>> the need to be asked for credentials. If allowed, this should be the
>> default.
>
>
> We can safely assume for now (through Spring) that builds will be executing
> on the same OpenShift instance that is hosting Almighty.
>
>>
>> = Next steps =
>> I will work on putting Build SPI proposal on paper and also on how we
>> should represent Build Service in core which is connected with it.
>>
>> There is also a userstory for next sprint on GitHub[3].
>>
>> I will appreciate your feedback!
>
>
> S,
> ALR
>>
>>
>> Regards,
>> Tomas
>>
>> [1] - https://www.redhat.com/archives/almighty-public/2016-September/ms
>> g00150.html
>> [2] - https://drive.google.com/file/d/0B10zSvDl_cuwMHZtTmR1RVIteGM/view
>>  (RH Only, but there is a screenshot in attached in [1])
>> [3] - https://github.com/almighty/almighty-core/issues/352
>>
>> _______________________________________________
>> almighty-public mailing list
>> almighty-public at redhat.com
>> https://www.redhat.com/mailman/listinfo/almighty-public
>
>
>
>
> --
> Red Hat Developer Programs Architecture
> @ALRubinger
>
> _______________________________________________
> almighty-public mailing list
> almighty-public at redhat.com
> https://www.redhat.com/mailman/listinfo/almighty-public
>