[almighty] Coverage service. How to proceed?

[Max Rydahl Andersen]

On 9 Sep 2016, at 18:38, Konrad Kleine wrote:

> Hi all,
>
> I've been investigating two coverage services on the web:
>
> https://codecov.io/gh/almighty/almighty-core/pull/235 and
> https://coveralls.io/github/almighty/almighty-core.
>
> The reason was to have some publicly facing coverage information for
> almighty-core (and possibly other repos under the almighty umbrella).

and getting first hand experience with services like this is good too 
since
we would most likely want to somehow have Almighty interact/link with 
such
in future.

> The pull requests to integrate codecov.io can be found here (
> https://github.com/almighty/almighty-core/pull/235) and this is how it
> looks in the PR once coverage was uploaded:
>
> The pull requests to integrate coveralls.io can be found here (
> https://github.com/almighty/almighty-core/pull/238) and this is how it
> looks in the PR once coverage was uploaded:

are these running on similar datasets ? the numbers on first glance 
looks quite different?


> While all of this is fine and dandy, I don't very much like the way 
> these
> services collect coverage information. Codecov.io collects all 
> *coverage*
> files which is simply wrong in our case and I don't want to adapt to 
> this
> system.
> We have very good reasons to have many coverage files (e.g. for
> different coverage modes and to not recollect coverage information 
> every
> time).

i'm not following why it is bad gathering all coverage files - can you 
elaborate ?
and who actually does the run of the build in this setup ? the service 
or is it grabbing it
from our jenkins ?

> Also, both service require a token to be submitted with the request 
> made to
> the service. Currently this is hard-coded in the PRs that I wrote just 
> to
> see some results. Those would have to be made invisible for others 
> which
> can become pretty hard in the current jenkins slave -> worker setup. 
> Even
> if the token is stored in environment variable it is easy to figure 
> out by
> simply modifying the cico_run_tests.sh script.

Yes, but isn't this why we now got accounts.yaml to store such tokens
and do not allow a PR to run before a human gives a +1 on it to avoid 
such things ?

Do you see any other way of handling it ?

> I don't see much benefit from having such a coverage service

IMO the fact we got it now and not a month from now I consider a win.

> and I'd like
> to see the Jenkins cobertura plugin in action (hopefully after the 
> next
> maintenance window). Then we don't have to mess with any external 
> service
> or token. I bet the devtools-test team has some ideas around providing 
> code
> coverage for the public.
>
> Please, let me know what to do with those PRs 235 and 238.

If they don't cause significant delays and we can get the token secured 
I think this is
better than just reading console output now.

btw. github will eventually detect tokens in our commits and disable 
their access - just in
case it suddenly stops working then you know why.

/max
http://about.me/maxandersen