[libvirt-jenkins-ci PATCH 2/7] lcitool: Configure ccache using environment variables

[Erik Skultety]

On Mon, Mar 30, 2020 at 01:18:49PM +0100, Daniel P. Berrangé wrote:
> On Mon, Mar 30, 2020 at 02:11:29PM +0200, Andrea Bolognani wrote:
> > On Mon, 2020-03-30 at 12:45 +0100, Daniel P. Berrangé wrote:
> > > On Fri, Mar 27, 2020 at 08:34:54PM +0100, Andrea Bolognani wrote:
> > > > +export CCACHE_MAXSIZE="2G"
> > >
> > > I was wondering what a good size for ccache would be. Is there any history
> > > to why we picked 2G ?  Having it too big didn't really matter for the
> > > Jenkins builders as it is kept local.  For GitLab the cache is downloaded
> > > at start of the job off cloud cstorage. So we want it large enough to fit
> > > a libvirt.git compile but small enough that outdated cruft gets purged
> > > reasonably quickly, so we don't waste time in GitLab CI downloading GB's
> > > of data that is no longer needed in the cache.
> > >
> > > NB, this is NOT an objection to this patch, as 2GB is a pre-existing value
> > > we used. Just want to know how we should consider tuning it in future.
> >
> > I think we just scaled it down (from the default of 5 GiB) so that
> > it would use most of the disk space that remained free in the VM's
> > 15 GiB disk, while leaving some leeway to allow for repositories to
> > grow. Nothing more scientific than that, I'm afraid.
>
> Ok, I'll see if I can get some usage stats out of ccache for our CI jobs on
> GitLab.
>
> > Note that, for VMs, we're building not just libvirt but a bunch of
> > other projects, so if we wanted to tweak it we'd have to take that
> > into account as well and not size it for just libvirt itself.
>
> True, but I imagine in terms of object size all the other projects probably
> barely reach 5% of the main libvirt.git build size, so likely lost in the
> noise.

The cache size that able to cover other libvirt-related projects which we
currently cover in jenkins is IMO a bit irrelevant in the context of gitlab,
because each project has their own namespace and their own pipeline, I've heard
from crosa (on CC) that inter-project pipeline dependencies exist to some
extend although such executions were not very reliable. Regarding the efforts
moving all workloads we can to gitlab from other sources of CI, solving the
current setup that we have in jenkins is going to be a challenge as I don't
think libvirt as a project on gitlab should run builds for other projects, they
can do it for themselves, we can and should only provide a base they can
consume for their upstream build tests.

Erik