[EnMasse] generated artifacts
Ulf Lilleengen
ulilleen at redhat.com
Wed Apr 12 07:42:56 UTC 2017
Hi all,
I have setup an EnMasse organization on bintray.com. Bintray provides
free artifact hosting for open source projects, and supports generic
artifacts, maven artifacts, rpm artifacts and more. It integrates easily
with github and travis.
With bintray, you upload artifacts using a generated API key (encrypted
in the travis config), and all members of the enmasse organization will
have permission to manage artifacts (though travis will upload both
snapshots and releases automatically):
https://bintray.com/enmasse
If you login using your github account, you might already have access to
the bintray configuration UI. If not, let me know and I'll grant access.
So far I've created a repo for generic artifacts and one for maven
artifacts:
https://bintray.com/enmasse/artifacts
https://bintray.com/enmasse/maven
The idea is that the 'artifacts' repository is the main repository,
where all build artifacts will be placed. These can then be fetched for
building docker docker images separately. It will contain artifacts for
c, java, node projects etc.
The maven repository will mainly be used for sharing maven dependencies
within enmasse components. The maven repository can be configured to
promote artifacts to jcentral maven repo automatically in the future if
we need that.
I have imported the github repositories into the bintray configuration.
Travis has 'native' support for uploading
artifacts(https://docs.travis-ci.com/user/deployment/bintray/), but I
found it a bit hard to parameterize the descriptor with the version from
the git tag. I made a script that is a bit more flexible:
https://github.com/EnMasseProject/travis-scripts/blob/master/upload-artifact.sh
For topic-forwarder, I've modified topic-forwarder to push the artifacts
to bintray as an initial example:
https://github.com/EnMasseProject/topic-forwarder/blob/master/.travis.yml#L13
I'll look more into using the native support as well since it provides
some nice metadata like git hashes.
You can download all artifacts here (I also did a test tag on
topic-forwarder so you can see both latest and <tag> artifacts):
https://dl.bintray.com/enmasse/artifacts/
We should also consider putting release artifacts and snapshots into
separate repositories (enmasse/releases, enmasse/snapshots for instance)
as the 'latest' tag will pollute any 'latest version' badges we might
want to use.
On 11. april 2017 20:15, Vanessa Busch wrote:
> Hi Ulf,
>
> It would be great if you could get access to that repo. I just had my
> 3 month review meeting with Kevin, and have been told to focus on the
> image files. If you can get the artifacts deployed to the repo that
> would really help.
>
> Thank you,
> Vanessa
>
> On Tue, Apr 11, 2017 at 1:59 PM, Ulf Lilleengen <ulilleen at redhat.com
> <mailto:ulilleen at redhat.com>> wrote:
>
> I'll admit the lack of storing artifacts have been bothering me as
> well. Travis assumes some external storage for that, such as amazon
> s3. We can also customize this and upload artifacts to an ftp server
> if we have one.
>
> I agree personal creds is suboptimal from a maintainability
> perspective, so if we can get a headless user, I think that would be
> ideal.
>
> Security wise, travis generates a private/public key pair for each
> repository, and the secrets in travis.yml are encrypted with the
> public key, and can only be decrypted with the travis private key on
> the travis servers. I agree it sounds a bit scary storing those in
> git, but attempting to break it would be the same as attempting to
> 'guess' the private key of anyone's public ssh key.
>
> In the longer term, we're going to remove those extra patches needed
> for qpid/proton, so that part would hopefully be temporary. At my
> previous company we used artifactory for all kinds of artifacts, c++
> and java. I will try to see if we can get access here:
> https://repo.jfrog.org/artifactory/webapp/#/home
> <https://repo.jfrog.org/artifactory/webapp/#/home>
>
>
>
> On 04/11/2017 06:56 PM, Rob Cernich wrote:
>
> The issue I have with travis, and I'm new to travis, is that it
> appears to be a continuous integration platform and not an
> actual build platform, so all artifacts are transient.
>
> I also agree with Vanessa, I'd be leery about placing my
> personal nexus creds in a travis.yaml just so I could deploy
> things to nexus. Maybe I don't fully understand things, but
> this seems like a gaping security hole (I probably don't, since
> folks are using this in production).
>
> Also, there's nothing that prevents us from continuing to use
> travis for the image builds. We're just talking about things
> like patched qpid/proton binaries. For the record, nexus is
> probably not the best place to store those anyway, as they don't
> exactly fit with maven.
>
>
> ----- Original Message -----
>
> Hi,
>
> Storing credentials is the same as for pushing docker images
> to docker
> hub. This is handled by encrypting the credentials in the
> .travis.yml.
> Is it possible to get a 'headless' user for deploying from CI?
>
> I'm a bit sceptical to just moving everything over to
> ci.wildfly just
> like that since travis works quite well today, and
> integrates so well
> with the github project. We can set it up in parallel and
> evaluate it
> (how long is our build queued, how stable is it etc.), but I
> don't think
> we should replace travis with it.
>
> On 11. april 2017 18:32, Vanessa Busch wrote:
>
> Hi Ulf,
>
> Assuming I get permissions to deploy to JBoss Nexus, I
> am concerned
> about having the builds doing the deploy, since they
> will need my
> credentials. There is something that doesn't seem right
> about that.
>
> If the ci-wildfly build works out, I thought it could
> build and hold
> our artifacts.
>
> Cheers,
> Vanessa
>
> On Tue, Apr 11, 2017 at 12:25 PM, Ulf Lilleengen
> <ulilleen at redhat.com <mailto:ulilleen at redhat.com>
> <mailto:ulilleen at redhat.com
> <mailto:ulilleen at redhat.com>>> wrote:
>
> Hi Vanessa,
>
> I think that being able to deploy artifacts to jboss
> nexus will be
> sufficient for the upstream project needs, as I've
> seen several
> projects having .zip artifacts published on jboss
> nexus. For
> downstream images, we can eventually use the
> messaging jenkins CI,
> or is this what you are looking into using wildfly for?
>
> On 11. april 2017 16:31, Vanessa Busch wrote:
>
> Hi Rob,
>
> A quick update, I have requested permissions,
> so I can deploy
> to JBoss
> Nexus as well as sonatype. I have been trying
> to find a contact
> for the
> ci.wildfly.org <http://ci.wildfly.org>
> <http://ci.wildfly.org> <http://ci.wildfly.org>,
> to see if it is possible to get
> credentials to add the Maas projects. My goal
> now, is to find an
> appropriate news group to post the question to.
>
> Thank you,
> Vanessa
>
> On Mon, Apr 10, 2017 at 4:18 PM, Rob Cernich
> <rcernich at redhat.com
> <mailto:rcernich at redhat.com> <mailto:rcernich at redhat.com
> <mailto:rcernich at redhat.com>>
> <mailto:rcernich at redhat.com
> <mailto:rcernich at redhat.com> <mailto:rcernich at redhat.com
> <mailto:rcernich at redhat.com>>>> wrote:
>
> Not sure if it's still available...most jobs
> make it look
> like it
> hasn't done anything for a year,
> but...https://hudson.jboss.org/hudson/
> <https://hudson.jboss.org/hudson/>
> <https://hudson.jboss.org/hudson/
> <https://hudson.jboss.org/hudson/>>
> <https://hudson.jboss.org/hudson/
> <https://hudson.jboss.org/hudson/>
> <https://hudson.jboss.org/hudson/
> <https://hudson.jboss.org/hudson/>>>
>
> It would be interesting to find out
> where/how folks are
> doing their
> community builds....which a quick google
> turns up:
> https://ci.wildfly.org (login as guest, no
> password).
> Interestingly, there's an artemis build there.
>
>
> ----- Original Message -----
> > On 04/10/2017 09:41 PM, Rob Cernich wrote:
> > >> Here is a description of the enmasse CI:
> > >>
>
>
> https://github.com/EnMasseProject/enmasse/tree/master/documentation/ci
> <https://github.com/EnMasseProject/enmasse/tree/master/documentation/ci>
>
> <https://github.com/EnMasseProject/enmasse/tree/master/documentation/ci
> <https://github.com/EnMasseProject/enmasse/tree/master/documentation/ci>>
>
>
> <https://github.com/EnMasseProject/enmasse/tree/master/documentation/ci
> <https://github.com/EnMasseProject/enmasse/tree/master/documentation/ci>
>
> <https://github.com/EnMasseProject/enmasse/tree/master/documentation/ci
> <https://github.com/EnMasseProject/enmasse/tree/master/documentation/ci>>>
> > >>
> > >> It basically does build & test ->
> pushes to docker hub
> -> triggers
> > >> systemtests which does a openshift
> deployment with
> enmasse +
> tests. I
> > >> would prefer if we could keep the
> upstream community
> bits built and
> > >> relased using travis + github.
> > >
> > > Given that we're talking about doing
> builds for the
> artifacts
> going into
> > > the images, I think it would be fine to
> use jenkins to
> build the
> artifacts
> > > used within the images (e.g. the patched
> versions of
> qpid/proton). This
> > > would create a clean separation between
> the images and the
> artifacts being
> > > installed in the images (which, btw,
> we'll need to do
> > > anyway
> once we get
> > > to productization). The only issue I
> see with doing this
> > > in
> travis is
> > > that travis doesn't store build
> artifacts, and there's
> no real
> place to
> > > upload these artifacts. So..hybrid
> approach, build the
> artifacts in
> > > jenkins; build the images in travis.
> Also, I'm guessing
> you can
> use a
> > > webhook with travis so you can invoke a
> build upon
> completion of the
> > > jenkins job.
> > >
> >
> > I agree it is nice to split artifact build
> from images, but
> hopefully we
> > could use jboss nexus for that.
> Alternatively if jboss
> nexus does not
> > support zip/tarballs, we have been granted
> a sonatype
> repository we
> > could start using:
> https://issues.sonatype.org/browse/OSSRH-25298
> <https://issues.sonatype.org/browse/OSSRH-25298>
> <https://issues.sonatype.org/browse/OSSRH-25298
> <https://issues.sonatype.org/browse/OSSRH-25298>>
>
> <https://issues.sonatype.org/browse/OSSRH-25298
> <https://issues.sonatype.org/browse/OSSRH-25298>
> <https://issues.sonatype.org/browse/OSSRH-25298
> <https://issues.sonatype.org/browse/OSSRH-25298>>>
> >
> > Using internal jenkins jobs for internal
> testing and
> productization is
> > ok, but we can't remove the upstream CI or
> have upstream
> depend on it.
> > The requirement for an upstream jenkins
> would be:
> >
> > * it must be publicly visible
> > * it must allow non-redhat employees part
> of the enmasse
> project to
> > configure and start jobs
> > * build/test github pull requests
> >
> > If there is a public jenkins instance we
> can use for this,
> then
> I'm not
> > against using that.
> >
> > >>
> > >> Setting up internal jenkins jobs to do
> these builds in
> addition
> would be
> > >> nice. We have a messaging CI that we
> can use for that,
> but I'm
> currently
> > >> awaiting some restructuring in the CI
> machines so that
> we can run
> > >> systemtests against an internal
> openshift instance.
> > >>
> > >>
> >
> >
> > --
> > Ulf Lilleengen
> >
>
>
>
> --
> Ulf
>
>
>
> --
> Ulf
>
>
>
> --
> Ulf Lilleengen
>
>
--
Ulf
More information about the enmasse
mailing list