[almighty] Build service and providers in Almighty - overview

Mon Oct 24 09:11:32 UTC 2016

Thanks for the feedback, Shoubhik.

Small clarification inline...

On Fri, 2016-10-21 at 23:18 +0530, Shoubhik Bose wrote:
> Thanks Tomas,
> The overview is detailed and inline with what I was thinking about ,
> and generally implemented.
> Responses inline ----
> On Oct 21, 2016 9:18 PM, "Tomas Nozicka" <tnozicka at redhat.com> wrote:
> >
> > Hi Shoubhik,
> >
> > you have a good point that I haven't described OAuth specifics in
> my
> > post, just mentioned it several times. That's mostly because I
> haven't
> > really decided yet on which approach to take and that mail was
> already
> > a long one to try to attack this issue.
> >
> > Continues inline...
> >
> > On Thu, 2016-10-20 at 19:02 +0530, Shoubhik Bose wrote:
> > > Hi Tomas,
> > > Very nice overview!
> > > I had a few thoughts on the auth tokens.
> > > I see it mentioned in a number of sections and wanted to share a
> generalized understanding:
> > > 1. ALM would need to be pre configured with OAuth application id
> + application secret for every external system we integrate with for
> builds. These could be read by alm-core as environment variables and
> passed on to the SPIs when needed. The environment variables can be
> set as Openshift Secrets.
> > > This allows ALM to identify itself when it talks to the external
> build system.
> > Well, in the approach you are correct. Although I don't
> > think "application id + application" is actually a requirement for
> > OAuth. It is just a way how some services provide you with OAuth
> > credentials to get token, but there are other ways. For example
> > OpenShift does not have those. (It has a concept of serviceaccounts
> > which by themselves provide you with a special long-term token to
> use
> > for authentication. But they need a namespace to be created in,
> setup
> > permissions for other namespaces; generally some non-trivial
> > management).
> >
> This sounds good. I agree, the applicationid and application secret
> are very external system specific and different systems have
> different ways of identifying who is accessing the specific external
> system.
> > The way I see it:
> >  - User will enter a URL of OAuth resource (like build environment)
> to
> > UI (e.g. openshift.com)
> >  - He will be redirected to openshift.com and asked for his regular
> > credentials
> >  - Almighty will get back a token that will be stored in
> configuration
> > with the URL
> >
> > The only issue with this approach is that the token might have
> limited
> > lifetime so when running build, Almighty shall detect expiry and
> ask
> > user to refresh token by redirecting him back to openshift.com. But
> we
> > should be prepared to do the same even if it would be long-term
> tokens
> > because they can still expire someday.
> >
> Yes this is the conventional procedure/approach.
> > There also a "backup" and more complicated way that we let user to
> > login by username and password and provider will setup long-term
> > service account on his behalf, but I would rather not go this way;
> at
> > least for now. (It may be needed in case of cross cluster
> deployments
> > but in a limited way.)
> Do you mean that this is not the usual OAuth workflow but that a
> username/password would eventually generate a server token?

Well, not really. It is a regular OAuth, the difference is with how you
setup the OAuth server. Let's be more concrete and use an example with
OpenShift here:

1. You can login to OpenShift's OAuth by using your regular credentials
(user: developer, password: developer); you will receive short-term
token and can do anything that user can
2. You can create service account with associated (long-term) token
(Alternative to appid+secret, which provide you with such token after
login in) and have some scoped (limited) access (it doesn't have to be
limited; scope can be "full")
3. You can use 1. to create 2. on users behalf and use 2. for ongoing
operations

Option 3. was the one I was referring to as "backup" and more
complicated solution.

We'll go with option 1 as our starting point.

> >
> > And there will be a default build environment, to the same
> OpenShift
> > instance as Almighty is running on, for which the above mentioned
> > authentication steps will result in "noop". (When user is already
> > logged in Almighty.)
> >
> Sounds good.
> > Yes, environment variables and secrets will be pasted to SPI
> provider
> > using CRUD operations. In case of OSPB this will be mapped to
> OpenShift
> > secrets and env variables for pipelines.
> >
> > > 2. With the help of of ALM UI , the user gives ALM authorization
> to manage builds on the external system. This step gives ALM an OAuth
> access token with which it is able to manage builds on the external
> system on the user's behalf.
> > > This access token allows ALM to do privileged stuff with the
> external system.
> > True.
> >
> > > Expiry :
> > > The expiry of access tokens should be set to fairly large as
> possible because the alm gets to specify the expiry during OAuth
> token generation procedure.
> > I don't think client is the one to choose and we can't influence
> > default values in target platform (OAuth server) like OpenShift
> Online.
> > We can do it for the default build environment which will be pre-
> > configured, but that's about it.
> >
> In some cases the client can ask for expiry times but in any case ,
> as you said there are always default values which could be banked
> upon.
> >
> > > We would need to have ways to notify the user in the UI when the
> token expires, gracefully.
> > Correct.
> >
> >
> > Regards,
> > Tomas
> >
> > >
> > > Let me know if this is what you intended to explain.
> > > Thanks
> > > Shoubhik
> > >
> > > On Oct 13, 2016 8:54 PM, "Tomas Nozicka" <tnozicka at redhat.com>
> wrote:
> > > > Hi all,
> > > >
> > > > this is a followup on Andy's initial mail[1] about this topic
> and
> > > > presents my view on what the Build Service (and providers)
> should look
> > > > like.
> > > >
> > > > Feel free to challenge any design here as you see fit; we are
> still in
> > > > early stages and figuring it out ourself.
> > > >
> > > > Andy published the generic design in his last email[1] at the
> highest
> > > > abstraction level and that stays the same. [2]
> > > >
> > > > Let's start by defining what Build Service is, because the name
> might
> > > > seem a bit confusing; at least it was for me when we joined
> this
> > > > project. Build Service should provide user with the ability to
> > > > transform repository code (e.g. git branch) into artifacts
> (like docker
> > > > images) and the ability to *deploy* them to target environment
> (e.g.
> > > > OpenShift). Build Service should be able to cover the whole
> CI/*CD*
> > > > story, not just builds.
> > > >
> > > > There is already a similar concept implemented in OpenShift
> called
> > > > *Pipelines* [3]. But, in ALM, we want to be more generic and
> provide
> > > > users with Build Service with plugable providers. That's why we
> are
> > > > creating Build SPI; to abstract the provider specifics away.
> Generally
> > > > speaking regular build (non-pipeline) job looks like a pipeline
> of 0
> > > > stages, which is a valid pipeline.
> > > >
> > > > Although there will be most probably different providers in the
> future,
> > > > we will start by implementing "OpenShift Build Provider" (OSBP)
> that
> > > > will use OpenShift's pipelines.
> > > >
> > > > There are 3 parts connected by Build SPI and Almighty public
> API:
> > > >  - ALM Core - Build Service
> > > >  - OSBP
> > > >  - ALM UI
> > > >
> > > > Good thing is that after we define the interfaces (mainly Build
> SPI)
> > > > those can be worked on in parallel.
> > > >
> > > > = ALM UI =
> > > > There is a lot of work done for visualizing pipelines in
> OpenShift's
> > > > Console. This is OSS and we could try to reuse those blocks.
> They will
> > > > most definitely need modifications because e.g. they read
> status
> > > > directly from OpenShift's API and that will be abstracted away
> by Build
> > > > SPI and ALM API in our case. Also I am not sure how well they
> visualize
> > > > pipelines without any stages. Also I remember Michael
> mentioning
> > > > Console uses Angular v1 and ALM UI v2.
> > > >
> > > > UI will also need to be able to indirectly ask user for
> credentials to
> > > > build and target/prod environment's (more generally to 1-N
> > > > environments) using OAuth2. ALM's core and build providers will
> > > > authenticate only by token produced by this (OAuth) step. And
> the rest
> > > > of those tokens will be given to build provider as secrets for
> > > > authentication to other environments (e.g. in case of cross-
> cluster
> > > > deployments).
> > > >
> > > >
> > > > = ALM Build Service =
> > > > This will be the service in ALM core that will issue calls
> against
> > > > Build SPI provider. How it will be represented, mainly what
> information
> > > > it needs to hold, will be strict superset of provider
> configuration.
> > > >
> > > >
> > > > = Build Providers =
> > > > Build provider will comunicate with Build Service through Build
> SPI
> > > > which abstracts provider specifics away. It will have several
> functions
> > > > like:
> > > >  - CRUD operations
> > > >  - StartBuild
> > > >  - CancelBuild
> > > >  - GetStages
> > > >  - GetLogs
> > > >  - and many more (this will be in Build SPI spec)
> > > > which can all be fit to work with both pipelines and regular
> build jobs
> > > > with the assumption that regular build job is pipeline of 0
> stages.
> > > >
> > > >
> > > > = Full picture =
> > > >  - ALM admin will add providers (register them) to ALM; or it
> will be
> > > > registered the way that ALM SPIs do... I haven looked into it
> yet:(
> > > >  - User will choose to add a Build Service to his project
> choosing
> > > > provider in UI
> > > >  - User will configure OAuth to his build environment [(URL)-
> >(token)]
> > > > and optionally other OAuth resources [(URL, name)->(name,
> token)] which
> > > > will be passed to build provider as secrets that can be used
> e.g. for
> > > > cross-cluster deployments in that build
> > > >    Since this will be done through OAuth, ALM shall never see
> user's
> > > > credentials.
> > > >  - Build Service instance is created in ALM core
> > > >  - Build Service uses provider to create BuildConfig instance
> passing
> > > > it necessary configuration like: token, secrets[(name, token)],
> > > > repository reference, ...
> > > >  - followed by many Build SPI calls...
> > > >
> > > > ALM core shall verify that all OAuth tokens are valid (not
> expired)
> > > > before calling Build SPI and refresh them with cooperation with
> ALM UI
> > > > if needed. (For the main token Build SPI will return
> unathorized, but
> > > > for the OAuth tokens in secrets the underlying technology does
> not
> > > > support that and build will be marked as failed otherwise. This
> might
> > > > be a common scenario since some tokens may last for about a day
> or so.)
> > > >
> > > > Also in case ALM will run on the same OpenShift instance as you
> setup
> > > > for your builds, or there will be a dedicated OpenShift
> instance for
> > > > ALM build set up with ALM as OpenShift's identity provider, you
> can be
> > > > authenticated to your build environment using your ALM token
> without
> > > > the need to be asked for credentials. If allowed, this should
> be the
> > > > default.
> > > >
> > > > = Next steps =
> > > > I will work on putting Build SPI proposal on paper and also on
> how we
> > > > should represent Build Service in core which is connected with
> it. 
> > > >
> > > > There is also a userstory for next sprint on GitHub[3].
> > > >
> > > > I will appreciate your feedback!
> > > >
> > > > Regards,
> > > > Tomas
> > > >
> > > > [1] - https://www.redhat.com/archives/almighty-public/2016-
> September/ms
> > > > g00150.html
> > > > [2]
> - https://drive.google.com/file/d/0B10zSvDl_cuwMHZtTmR1RVIteGM/view
> > > >  (RH Only, but there is a screenshot in attached in [1])
> > > > [3] - https://github.com/almighty/almighty-core/issues/352
> > > >
> > > > _______________________________________________
> > > > almighty-public mailing list
> > > > almighty-public at redhat.com
> > > > https://www.redhat.com/mailman/listinfo/almighty-public
> > > >