<div dir="ltr"><div dir="ltr"><div>Hi Tristan,</div><div> </div><div>Thanks for this great recap. I updated the related Taiga epic today according to the README TODO list.</div><div>Then we now have 4 more stories to groom in : <a href="https://tree.taiga.io/project/morucci-software-factory/epic/2077">https://tree.taiga.io/project/morucci-software-factory/epic/2077</a></div><div> </div><div>Cheers,</div><div>Fabien </div></div></div> <div class="gmail_quote"><div dir="ltr" class="gmail_attr">On Wed, Mar 13, 2019 at 8:30 AM Tristan Cacqueray <<a href="mailto:tdecacqu@redhat.com">tdecacqu@redhat.com</a>> wrote: </div><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex"> Hello, This sprint we worked on a new set of distro-jobs to investigate a new rpm build pipeline. The goal is to unify the <a href="http://review.rdoproject.org" rel="noreferrer" target="_blank">review.rdoproject.org</a> rdoinfo, softwarefactory sfinfo and dci dci-config jobs. All these jobs do the same thing: build rpm on new change, enable integration tests, and publish them in the gate pipeline. Here is the results of this investigation: # Add a new distroinfo configuration endpoint in managesf resources The first idea is to remove duplication of information and keeps package information in the existing resources definition. To do that we plan on adding a "distroinfo" resource type: ``` resources: distroinfos: # the distro name tdpw: projects: # the project to collect package information - tdpw distroinfo: releases: - name: master branch: rpm-master # custom templates for 'source-repositories' configuration package-configs: internal-package: name: "%(project)s" upstream: <a href="https://softwarefactory-project.io/r/tdpw/%(project)s.git" rel="noreferrer" target="_blank">https://softwarefactory-project.io/r/tdpw/%(project)s.git</a> distgit: <a href="https://softwarefactory-project.io/r/tdpw/%(project)s-distgit.git" rel="noreferrer" target="_blank">https://softwarefactory-project.io/r/tdpw/%(project)s-distgit.git</a> internal-package-with-spec-included: name: "%(project)s" upstream: <a href="https://softwarefactory-project.io/r/tdpw/%(project)s.git" rel="noreferrer" target="_blank">https://softwarefactory-project.io/r/tdpw/%(project)s.git</a> distgit: <a href="https://softwarefactory-project.io/r/tdpw/%(project)s.git" rel="noreferrer" target="_blank">https://softwarefactory-project.io/r/tdpw/%(project)s.git</a> external-package-openstack: name: "%(project)s" upstream: <a href="https://git.openstack.org/openstack/%(project)s.git" rel="noreferrer" target="_blank">https://git.openstack.org/openstack/%(project)s.git</a> distgit: <a href="https://softwarefactory-project.io/r/tdpw/%(project)s-distgit.git" rel="noreferrer" target="_blank">https://softwarefactory-project.io/r/tdpw/%(project)s-distgit.git</a> ``` Then in the project source-repository list we could have a 'package-config' option so that the endpoint would be able to generate the distroinfo file with all the package automatically included, similarly to what we do for zuul tenant config. # dnrl-preprare job provided by sfconfig After deployment, sfconfig configures a dlrn-base abstract job that will use the dlrn-prepare role with these variables: ``` dlrn_url: <a href="https://sftests.com/dlrn/" rel="noreferrer" target="_blank">https://sftests.com/dlrn/</a> dlrn_api_url: <a href="https://sftests.com/dlrn-worker-api/" rel="noreferrer" target="_blank">https://sftests.com/dlrn-worker-api/</a> dlrn_config_dir: <a href="http://sftests.com/config/dlrn" rel="noreferrer" target="_blank">sftests.com/config/dlrn</a> dlrn_config_url: <a href="https://sftests.com/manage/v2/configurations/dlrn" rel="noreferrer" target="_blank">https://sftests.com/manage/v2/configurations/dlrn</a> ``` The goal of dlrn-prepare is to setup the distroinfo.yml and dlrn.ini on the test instances. # dlrn-build and dlrn-publish job pipeline The distro-jobs repository contains a couple of abstract jobs. The dlrn-build definition looks like this: ``` - job: name: dlrn-build description: Build project's package. parent: dlrn-base run: playbooks/dlrn/build.yaml post-run: playbooks/dlrn/fetch.yaml provides: dlrn-repo requires: dlrn-repo ``` It uses the new provides/requires zuul artifact logic. The idea is that in the case of depends-on, we want to be able to use the rpm built by the change ahead without rebuild them in each queue item. The enabling task is: ``` - name: Add intermediary repositories from changes ahead blockinfile: path: "{{ dlrn_data }}/{{ dlrn_project }}.cfg" marker: '' insertafter: '# SPECULATIVE REPO HERE' content: | {% for artifact in zuul.artifacts|default([]) %}{% if <a href="http://artifact.name" rel="noreferrer" target="_blank">artifact.name</a> == 'dlrn-repo' %} [{{ artifact.project }}-{{ artifact.change }}-{{ artifact.patchset }}] name={{ artifact.project }}-{{ artifact.change }}-{{ artifact.patchset }} enabled=1 baseurl={{ artifact.url }}/current gpgcheck=0 {% endif %}{% endfor %} ``` # the project interface The internal details of distro-jobs and configuration are still being explored, but the end user interface should looks like this: * Defines the distro information in the resources * Add dlrn-build and dlrn-publish job to project's pipelines. Here is an example distribution pipeline configurations: ``` --- # Distro jobs implementation - job: name: tdpw-dlrn-build parent: dlrn-build vars: dlrn_project: tdpw - job: name: tdpw-dlrn-publish parent: dlrn-publish vars: dlrn_project: tdpw dlrn_user: username: tdpw password: secret # Distro ci jobs - job: name: tdpw-ci-functional requires: dlrn-repo run: playbooks/functional.yaml # Ci pipelines - project-template: name: tdpw-jobs check: jobs: - tdpw-dlrn-build - tdpw-ci-functional: dependencies: - tdpw-dlrn-build gate: queue: tdpw jobs: - tdpw-dlrn-build - tdpw-ci-functional: dependencies: - tdpw-dlrn-build - wait-for-changes-ahead: dependencies: - tdpw-ci-functional - tdpw-dlrn-publish: dependencies: - wait-for-changes-ahead release: jobs: - tdpw-dlrn-publish ``` # Automatic dependencies bump Time ran out, but it seems like we should also be able to leverage the git-drivers or the gerrit-drivers to also run the DLRN service inside a zuul jobs. For example, when a new tag is created in one the upstream project, a dlrn-propose job could propose a review to bump the version number automatically. # Example implementations We prototyped those changes with the tdpw project ([t]est [d]istribution [p]ackaging [w]orkflow): The dlrn-prepare job: <a href="https://softwarefactory-project.io/r/15206" rel="noreferrer" target="_blank">https://softwarefactory-project.io/r/15206</a> The distro-jobs: <a href="https://softwarefactory-project.io/r/15207" rel="noreferrer" target="_blank">https://softwarefactory-project.io/r/15207</a> The tdpw ci pipelines: <a href="https://softwarefactory-project.io/r/15208" rel="noreferrer" target="_blank">https://softwarefactory-project.io/r/15208</a> Todos are tracked in the README of distro-jobs. Regards, -Tristan -----BEGIN PGP SIGNATURE----- iQEzBAEBCAAdFiEE6xA96LXmnmMcb/F5IrmgXJJcxdgFAlyIsZIACgkQIrmgXJJc xdg9Zwf/TIzaizv8Of1iEpzul+plR5e6+bzvn+73Bg2B0zL4GewqH4Xix5yCm322 8Hnym6GITT2XqoFaEeg5Vvq69AXGJ29YN0p+LaSbKD0txuIA1jUNJ17zEbISV1eC TSLF7HxcMCH1OvLg46Njudfd71/RjEKkrNQyTUqsEnTUJpIFXhkhHLgZwiApgtvX OL0uzldgMxL5DfM8mSgE1SsvtKybyJjLba445rXyYIj+KE4Y8aAJSy8kbeV2sPfV ki9H0attqiMv2WD2I88ODg/J0MNTA7vnLfoQwnxA43fHUw8bctE0qxNSlkJy0/Hh iLVaK0qHbbSpl3tIFFN8tpKqnd2l/g== =pDmG -----END PGP SIGNATURE-----_______________________________________________ Softwarefactory-dev mailing list <a href="mailto:Softwarefactory-dev@redhat.com" target="_blank">Softwarefactory-dev@redhat.com</a> <a href="https://www.redhat.com/mailman/listinfo/softwarefactory-dev" rel="noreferrer" target="_blank">https://www.redhat.com/mailman/listinfo/softwarefactory-dev</a> </blockquote></div>