[almighty] Build service and providers in Almighty - overview

Michael Kleinhenz kleinhenz at redhat.com
Thu Oct 20 11:26:49 UTC 2016 

Very good kickoff thoughts, thanks!

For the UI integration: I think they are an essential component and at
the same time the area with the highest tech risk, because we have to
do some very specific R&D to find a way how we can integrate 3rd party
UI components. We should go fo this asap and define trust levels and
specific tech requirements for this. I consider the UI part as an
integral part of the SPI definition. For starters, Facebook has a some
similar req's for Facebook Apps that are integrated into the Facebook
page. They have found a solution that is very well documented on how
they do that.

-- Michael

On Thu, Oct 20, 2016 at 9:20 AM, Andrew Lee Rubinger <alr at redhat.com> wrote:
> Great; for all areas that we're in alignment I'll leave alone, and make only
> additional notes for the open items.
>
> On Wed, Oct 19, 2016 at 6:22 PM, Tomas Nozicka <tnozicka at redhat.com> wrote:
>>
>> Comments inline.
>>
>> On Po, 2016-10-17 at 10:38 +0200, Andrew Lee Rubinger wrote:
>> > Overall I think this is a very good summary and start to the
>> > discussion, as well as the userstory in [3].  I have added some color
>> > and clarifications here in the comments.
>> >
>> > On Thu, Oct 13, 2016 at 5:26 PM, Tomas Nozicka <tnozicka at redhat.com>
>> > wrote:
>> > > Hi all,
>> > >
>> > > this is a followup on Andy's initial mail[1] about this topic and
>> > > presents my view on what the Build Service (and providers) should
>> > > look
>> > > like.
>> > >
>> > > Feel free to challenge any design here as you see fit; we are still
>> > > in
>> > > early stages and figuring it out ourself.
>> > >
>> > > Andy published the generic design in his last email[1] at the
>> > > highest
>> > > abstraction level and that stays the same. [2]
>> >
>> > Great; if we start to see holes in this design please raise the flag
>> > as soon as possible. :)  We'll make the necessary changes.
>> >
>> > > Let's start by defining what Build Service is, because the name
>> > > might
>> > > seem a bit confusing; at least it was for me when we joined this
>> > > project. Build Service should provide user with the ability to
>> > > transform repository code (e.g. git branch) into artifacts (like
>> > > docker
>> > > images) and the ability to *deploy* them to target environment
>> > > (e.g.
>> > > OpenShift). Build Service should be able to cover the whole CI/*CD*
>> > > story, not just builds.
>> >
>> > +1; "Build" is overloaded, and "CI/CD" can be misleading.  I'm open
>> > to changing the name of this, so long as it's clear that's the goal
>> > here is to take source from some repo at some revision and *do
>> > something* with it.  If this sounds unscoped and generic, that's
>> > because pipelines by their very nature are as well. :)
>> Agreed. Not sure about proper name for it, though. Generally this is
>> just scriptable source (revision) transformation. (Into artifact,
>> action, or both.)
>>
>> >
>> > > There is already a similar concept implemented in OpenShift called
>> > > *Pipelines* [3]. But, in ALM, we want to be more generic and
>> > > provide
>> > > users with Build Service with plugable providers. That's why we are
>> > > creating Build SPI; to abstract the provider specifics away.
>> > > Generally
>> > > speaking regular build (non-pipeline) job looks like a pipeline of
>> > > 0
>> > > stages, which is a valid pipeline.
>> >
>> > Of 1 stage, Stage 0, right? :)
>> Actually I believe 0 stages is right. Here is my reasoning:
>> According to Jenkins if you do pipeline without any explicit stages and
>> run it (successfully), its UI tells you that there are no stages in the
>> pipeline == 0 stages. There are only actions not related to any stage.
>> Stage seems to be only explicitly marked group of actions by "stage"
>> instruction. Assuming you have some actions, then stage instruction
>> followed by some actions, you end up with 1 stage and some instructions
>> that do not belong to any stage.
>> So I would say there can be pipeline without stages == pipeline of 0
>> stages.
>
>
> OK, good argument; let's use the "0 stage" definition you suggest here.
>
>>
>>
>> >
>> > > Although there will be most probably different providers in the
>> > > future,
>> > > we will start by implementing "OpenShift Build Provider" (OSBP)
>> > > that
>> > > will use OpenShift's pipelines.
>> >
>> > ...as fulfilled via Jenkins Pipeline.
>> >
>> > > There are 3 parts connected by Build SPI and Almighty public API:
>> > >  - ALM Core - Build Service
>> > >  - OSBP
>> > >  - ALM UI
>> > >
>> > > Good thing is that after we define the interfaces (mainly Build
>> > > SPI)
>> > > those can be worked on in parallel.
>> >
>> > Excellent; that's the dream here.  That we define:
>> >
>> > 1) The Build SPI contract between Almighty and a Build Provider
>> > 2) The Build Domain objects and Service operations known by Almighty
>> > 3) How 2) plugs into the Almighty Public API for use by the UI
>> >
>> > ...that gets groups working independently.
>> +1
>>
>> >
>> > > = ALM UI =
>> > > There is a lot of work done for visualizing pipelines in
>> > > OpenShift's
>> > > Console. This is OSS and we could try to reuse those blocks. They
>> > > will
>> > > most definitely need modifications because e.g. they read status
>> > > directly from OpenShift's API and that will be abstracted away by
>> > > Build
>> > > SPI and ALM API in our case. Also I am not sure how well they
>> > > visualize
>> > > pipelines without any stages. Also I remember Michael mentioning
>> > > Console uses Angular v1 and ALM UI v2.
>> >
>> > Now that this discussion is kicked off here by Tomas, I'll raise the
>> > question with the OpenShift Dev List for exploring what opportunities
>> > we have to share or consume code, to the extent that is technically
>> > feasible.
>> +1
>>
>> >
>> > > UI will also need to be able to indirectly ask user for credentials
>> > > to
>> > > build and target/prod environment's (more generally to 1-N
>> > > environments) using OAuth2. ALM's core and build providers will
>> > > authenticate only by token produced by this (OAuth) step. And the
>> > > rest
>> > > of those tokens will be given to build provider as secrets for
>> > > authentication to other environments (e.g. in case of cross-cluster
>> > > deployments).
>> >
>> > Or we'll need to have this info tied into the concept of an Almighty
>> > user account.  We need to flesh this out some more I believe; at what
>> > point do we gain auth to the OpenShift Online "Environment" for
>> > deployment?
>> We can have user configure several OAuth resources at "user level" and
>> have him choose on project level: which of those should be used for
>> "Build" (dev) environment and which others he wants to access for
>> cross-cluster deployment (prod).
>
>
> At the "Almighty User Level"?  If so +1.  That gives us some mappings to
> choose from when mapping identityProvider>targetEnvironment.
>
>>
>> >
>> > > = ALM Build Service =
>> > > This will be the service in ALM core that will issue calls against
>> > > Build SPI provider. How it will be represented, mainly what
>> > > information
>> > > it needs to hold, will be strict superset of provider
>> > > configuration.
>> >
>> > Almighty team gets to define this piece, assuming it knows nothing of
>> > the provider (ie. Jenkins or OpenShift) carrying out the tasks
>> > requested.  It's our abstraction layer.
>> +1
>>
>> >
>> > > = Build Providers =
>> > > Build provider will comunicate with Build Service through Build SPI
>> > > which abstracts provider specifics away. It will have several
>> > > functions
>> > > like:
>> > >  - CRUD operations
>> > >  - StartBuild
>> > >  - CancelBuild
>> > >  - GetStages
>> > >  - GetLogs
>> > >  - and many more (this will be in Build SPI spec)
>> >
>> > +1; we need to account for all build artifacts including logs, test
>> > runs, coverage and static analysis reports, etc.
>> >
>> > > which can all be fit to work with both pipelines and regular build
>> > > jobs
>> > > with the assumption that regular build job is pipeline of 0 stages.
>> >
>> > 1 stage. :P
>> most probably 0 since there can be pipeline without any stages :P
>> (as explained above)
>>
>> >
>> > > = Full picture =
>> > >  - ALM admin will add providers (register them) to ALM; or it will
>> > > be
>> > > registered the way that ALM SPIs do... I haven looked into it yet:(
>> >
>> > Aslak is on this "Service Registry" piece.
>> >
>> > >  - User will choose to add a Build Service to his project choosing
>> > > provider in UI
>> >
>> > For this first release, we can assume there's only one, the OpenShift
>> > one.
>> >
>> > >  - User will configure OAuth to his build environment [(URL)-
>> > > >(token)]
>> > > and optionally other OAuth resources [(URL, name)->(name, token)]
>> > > which
>> > > will be passed to build provider as secrets that can be used e.g.
>> > > for
>> > > cross-cluster deployments in that build
>> >
>> > This might be part of the "Environments" definition.  I'll get us an
>> > answer here from PM.
>> Also possible.
>>
>> >
>> > >    Since this will be done through OAuth, ALM shall never see
>> > > user's
>> > > credentials.
>> > >  - Build Service instance is created in ALM core
>> >
>> > Build Service instance per...Almighty instance?  Almighty Project?
>> > Almighty User?  Depends on what state the service itself is carrying,
>> > and what state we pass along as parameters in a request.  WDYT here?
>> >
>> > Seems to me we could get away with a Build Service instance per
>> > provider, shared by the Almighty system, if requests are scoped to
>> > the Almighty project in question.
>> Yeah, we can probably have one Build Service per Almighty instance if
>> the state for it will be represented by project Build Config. (And
>> provider ID can be part of the state in BuildConfig.)
>>
>> Build Provider will then be 1 per Almighty instance (or even more, in
>> theory) since it will be RESTful (stateless) service.
>
>
> For resource sanity and in service of keeping the architecture as stateless
> as we can (pushing the state into requests), this makes sense to me so far,
> pending technical arguments why this can't be done.  Good, let's go down
> this path for now.
>
>>
>>
>> >
>> > >  - Build Service uses provider to create BuildConfig instance
>> > > passing
>> > > it necessary configuration like: token, secrets[(name, token)],
>> > > repository reference, ...
>> > >  - followed by many Build SPI calls...
>> > >
>> > > ALM core shall verify that all OAuth tokens are valid (not expired)
>> > > before calling Build SPI and refresh them with cooperation with ALM
>> > > UI
>> > > if needed. (For the main token Build SPI will return unathorized,
>> > > but
>> > > for the OAuth tokens in secrets the underlying technology does not
>> > > support that and build will be marked as failed otherwise. This
>> > > might
>> > > be a common scenario since some tokens may last for about a day or
>> > > so.)
>> > >
>> > > Also in case ALM will run on the same OpenShift instance as you
>> > > setup
>> > > for your builds, or there will be a dedicated OpenShift instance
>> > > for
>> > > ALM build set up with ALM as OpenShift's identity provider, you can
>> > > be
>> > > authenticated to your build environment using your ALM token
>> > > without
>> > > the need to be asked for credentials. If allowed, this should be
>> > > the
>> > > default.
>> >
>> > We can safely assume for now (through Spring) that builds will be
>> > executing on the same OpenShift instance that is hosting Almighty.
>> For OSBP it does make any difference. Maybe Build Service and UI won't
>> actually have to ask user to set up OAuth and just generate it
>> internally to the same OpenShift instance for now.
>>
>>
>> >
>> > > = Next steps =
>> > > I will work on putting Build SPI proposal on paper and also on how
>> > > we
>> > > should represent Build Service in core which is connected with it.
>> > >
>> > > There is also a userstory for next sprint on GitHub[3].
>> > >
>> > > I will appreciate your feedback!
>> >
>> > S,
>> > ALR
>> > >
>> > > Regards,
>> > > Tomas
>> > >
>> > > [1] - https://www.redhat.com/archives/almighty-public/2016-Septembe
>> > > r/ms
>> > > g00150.html
>> > > [2] - https://drive.google.com/file/d/0B10zSvDl_cuwMHZtTmR1RVIteGM/
>> > > view
>> > >  (RH Only, but there is a screenshot in attached in [1])
>> > > [3] - https://github.com/almighty/almighty-core/issues/352
>> > >
>> > > _______________________________________________
>> > > almighty-public mailing list
>> > > almighty-public at redhat.com
>> > > https://www.redhat.com/mailman/listinfo/almighty-public
>> > >
>> >
>> >
>> >
>> > --
>> > Red Hat Developer Programs Architecture
>> > @ALRubinger
>
>
>
>
> --
> Red Hat Developer Programs Architecture
> @ALRubinger
>
> _______________________________________________
> almighty-public mailing list
> almighty-public at redhat.com
> https://www.redhat.com/mailman/listinfo/almighty-public
>



-- 
Michael Kleinhenz
Principal Software Engineer

Red Hat Deutschland GmbH
Werner-von-Siemens-Ring 14
85630 Grasbrunn
Germany

RED HAT | TRIED. TESTED. TRUSTED.
Red Hat GmbH, www.de.redhat.com,
Registered seat: Grasbrunn, Commercial register: Amtsgericht München,
HRB 153243,
Managing Directors: Paul Argiry, Charles Cachera, Michael Cunningham,
Michael O'Neill

More information about the almighty-public mailing list