[almighty] CI infrastructure update

[Pavol Pitonak]

Hi,

yes, it would require pushing the built image to some docker registry and
then download it. In the long term, there should be some local (Maven, NPM,
Docker) cache but we don't have it yet.

1. Can you estimate how much overhead it is upload and download image?
2. How often does this image change?
3. What would be better - leaving process as is now but have multiple
dynamically provisioned Jenkins slaves (cache will be lost now and then) or
changing the process as I suggested in original email? My guess is that
uploading image once and downloading it frequently is cheaper than building
Docker image locally relatively frequently.

Regards,
Pavol

On Mon, Aug 8, 2016 at 3:42 PM, Konrad Kleine <kkleine at redhat.com> wrote:

> Hi Pavol,
>
> thank you for the quick and detailed update on your plans.
>
> You asked us to separate the building of docker images from compiling with
> this image. Let's assume that we could do that. How will the other Jenkins
> slaves with docker installed know about the image that we've built? Will
> they share storage?
>
> I'm not saying that we should do it but we could push the image to some
> docker registry and download it to the slave that needs it. The overhead of
> this approach would be significant.
>
> Regards,
> Konrad
>
> On Mon, Aug 8, 2016 at 3:32 PM, Pavol Pitonak <ppitonak at redhat.com> wrote:
>
>> Hi,
>>
>> I'd like to inform team about status quo and future plans regarding our
>> continuous integration infrastructure.
>>
>> Almighty team uses internal instance of Jenkins hosted on Red Hat
>> infrastructure internally referred to as Central CI, CCI or Central CI
>> Jenkins. Contact person is Pavol Srna, Martin Malina or me.
>>
>> Jenkins master is managed by ops team and runs in a VM, slaves are
>> managed by QE team and are provisioned in OpenStack 7 instance. At the
>> moment we have resource quota for provisioning 10 relatively powerful (4
>> vcpus, 8gb ram) slaves that are shared by Almighty and JBoss Tools teams.
>>
>> All Almightly Jenkins jobs use Docker but unfortunately there is only 1
>> Jenkins slave with Docker at the moment.
>>
>> Short term plan (possible to implement immediately):
>> * Unify all RHEL7 slaves to just one type that would have Docker
>> installed. There is one issue, though. Jenkins job for almighty-core builds
>> a custom Docker image which takes quite a lot of time therefore we decided
>> to provision a dedicated Jenkins slave so that Docker cache wouldn't be
>> pruned too often.
>>
>> If we want to use our resources in the most effective way possible, we
>> need to provision slaves dynamically (up to some limit) and then destroy
>> them when not used. I would need you to split building this interim Docker
>> image and building Almighty-core into two separate jobs. Then almighty-core
>> job could run on whatever Jenkins slave relatively fast.
>>
>> After you do it, we would just update Jenkins configuration for slaves
>> provisioning (5 minutes of work).
>>
>> Long term plan (no ETA right now):
>> * We asked for our quota to be increased significantly, should at least
>> double at the beginning.
>> * Jenkins master will be upgraded from RHEL6 to RHEL7, no direct impact
>> on Almighty team.
>> * Jenkins will be upgraded to newer LTS version (1.651.3). There was
>> interest in new Jenkins Pipeline plugins - this plugin will be supported in
>> this new version of Jenkins.
>> * Define requirements for Jenkins slaves - OS, minimum CPUs, minimum RAM,
>> installed software - both for what is required now and what will be
>> necessary in the future (e.g. testing on multiple operating systems,
>> multiple versions of ???)
>> * Right now community cannot see Jenkins builds because the Jenkins runs
>> on internal infrastructure. It's technically possible to publish results to
>> a public Jenkins instance. ATM it's not clear if this is desired and how to
>> implement.
>> * Decide if a dedicated Jenkins master (not shared with other projects)
>> would be more suitable in the future (IMHO would be too much overhead for
>> team at the moment)
>>
>> P.S. We installed Warnings Jenkins plugin
>>
>> Regards,
>> Pavol
>>
>>
>> _______________________________________________
>> almighty-public mailing list
>> almighty-public at redhat.com
>> https://www.redhat.com/mailman/listinfo/almighty-public
>>
>>
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://listman.redhat.com/archives/almighty-public/attachments/20160809/959cc650/attachment.htm>