<div dir="ltr"><div>Hi Konstantin,</div><div><br></div><div>surprisingly today i was able to reproduce the issue and was able to find the root cause of the problem.</div><div>Here's the PR that should fix the upload issue <a href="https://github.com/pulp/pulp_docker/pull/418">https://github.com/pulp/pulp_docker/pull/418</a></div><div><div><div dir="ltr" class="gmail_signature" data-smartmail="gmail_signature"><div dir="ltr"><div><div dir="ltr"><br><br>--------<br>Regards,<br><br>Ina Panova<br>Senior Software Engineer| Pulp| Red Hat Inc.<br><br>"Do not go where the path may lead,<br> go instead where there is no path and leave a trail."<br></div></div></div></div></div><br></div></div><br><div class="gmail_quote"><div dir="ltr" class="gmail_attr">On Tue, Sep 17, 2019 at 11:58 AM Konstantin M. Khankin <<a href="mailto:khankin.konstantin@gmail.com">khankin.konstantin@gmail.com</a>> wrote:<br></div><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex"><div dir="ltr"><font size="4">Sorry for necroposting. I'm not sure if I still have the same issue, but I have the same symptoms. Could you please have a look and advice? Thanks!</font><div><font size="4"><br></font></div><div><font size="4">Whenever I try to create and upload another version of a container to the same repo, I get an error message like this:</font></div><div><br></div><div><font face="monospace">$ /usr/bin/pulp-admin docker repo uploads upload --repo-id gsk_postfix -f /tmp/postfix.tar<br></font></div><div><font face="monospace">...</font></div><div><font face="monospace">Uploading: postfix.tar<br>[==================================================] 100%<br>283176960/283176960 bytes<br>... completed<br><br>Importing into the repository...<br>This command may be exited via ctrl+c without affecting the request.<br><br><br>[-]<br>Running...<br><br>Task Failed<br><br>The importer docker_importer indicated a failed response when uploading<br>docker_manifest unit to repository gsk_postfix. Summary: 'Cannot resolve field<br>"digest"'. Details: '{}' </font><br></div><div><font size="4"><br></font></div><div><font size="4">I run pulp 2.20.1 and use buildah 1.9.0 to build container images. First image in the repository uploads fine, but the second always fails.</font></div><div><font size="4"><br></font></div><div><font size="4">Current repo status:</font></div><div><font face="monospace">Id: gsk_postfix<br>Display Name: None<br>Description: None<br>Content Unit Counts: <br> Docker Blob: 4<br> Docker Manifest: 1<br> Docker Tag: 1<br></font></div><div><font face="monospace"><br></font></div><div><font face="monospace"># pulp-admin docker repo search image --repo-id gsk_postfix<br></font></div><div><font face="monospace"><returns nothing></font></div><div><font face="monospace"><br></font></div><div><font face="monospace"># pulp-admin docker repo search manifest --repo-id gsk_postfix<br>Created: 2019-08-20T22:58:50Z<br>Metadata: <br> Config Layer: sha256:d2b497c83fe099624567046432468d84a5fdb49139b5192c9b332f0b10784695<br> Digest: sha256:d17067c71cb229f703b0f43ecd8a85c55995f3007b99570f580f9f6ebc5dbccc<br> Downloaded: True<br> Fs Layers: <br> Blob Sum: sha256:b05580fca2f9aabb2d8fa975b29146c9147c8418e559f197c54a4fac04babb95<br> Layer Type: application/vnd.docker.image.rootfs.diff.tar.gzip<br> Size: 210204160<br> Blob Sum: sha256:204cafbfdb1d31cb427478102377a99ceccfd7cc7be19cc191568fb9008fd4af<br> Layer Type: application/vnd.docker.image.rootfs.diff.tar.gzip<br> Size: 12288<br> Blob Sum: sha256:a85e1551e989545ca081c9ed0584e2db535a1484b91060eed88a17991e4b25cf<br> Layer Type: application/vnd.docker.image.rootfs.diff.tar.gzip<br> Size: 72946688<br> Pulp User Metadata: <br> Schema Version: 2<br>Repo Id: gsk_postfix<br>Unit Id: 000c0b4a-fb98-4ca3-a3ec-3b2a42521949<br>Unit Type Id: docker_manifest<br>Updated: 2019-08-20T22:58:50Z<br></font></div><div><font face="monospace"><br></font></div><div><font face="monospace"># pulp-admin docker repo search tag --repo-id gsk_postfix<br>Created: 2019-08-20T22:58:55Z<br>Metadata: <br> Manifest Digest: sha256:d17067c71cb229f703b0f43ecd8a85c55995f3007b99570f580f9f6ebc5dbccc<br> Manifest Type: image<br> Name: latest<br> Pulp User Metadata: <br> Repo Id: gsk_postfix<br> Schema Version: 2<br>Repo Id: gsk_postfix<br>Unit Id: 26c40bb1-4f3b-438f-8f50-cc5afe6baeb6<br>Unit Type Id: docker_tag<br>Updated: 2019-08-20T22:58:55Z</font><br></div><div><br></div><div><font size="4">To build and upload containers, I wrote a simple script. Here is the command sequence:</font></div><div><font face="monospace">### STEP 1: Build an image from Dockerfile.<br>$ /usr/bin/buildah bud -t gsk_postfix .<br></font></div><div><font face="monospace">...</font></div><div><font face="monospace">Writing manifest to image destination<br>Storing signatures<br>db008cf80da01afe8ca11898e566279a7765e682bf56070857d40ffd469e2486<br></font></div><div><font face="monospace">### STEP 2: Get the image digest.<br>$ /usr/bin/buildah images --quiet --digests --no-trunc localhost/gsk_postfix:latest<br>sha256:db008cf80da01afe8ca11898e566279a7765e682bf56070857d40ffd469e2486<br></font></div><div><font face="monospace">### STEP 3: Pull blobs from the storage.<br>$ /usr/bin/skopeo copy --format v2s2 containers-storage:localhost/gsk_postfix:latest dir:/tmp/tmp.NYhxiiOsOe<br></font></div><div><font face="monospace">Getting image source signatures<br>Copying blob b05580fca2f9 done<br>Copying blob 8d136bb340b2 done<br>Copying blob 7a406e7184ab done<br>Copying config db008cf80d done<br>Writing manifest to image destination<br>Storing signatures<br></font></div><div><font face="monospace">### STEP 4: Tar the archive.<br>$ tar cf /tmp/postfix.tar *<br></font></div><div><font face="monospace">### STEP 5: Upload the archive to the repo.<br>$ /usr/bin/pulp-admin docker repo uploads upload --repo-id gsk_postfix -f /tmp/postfix.tar<br></font></div><div><font face="monospace"><failure here></font></div><div><br></div><div><font size="4" face="arial, sans-serif">This happens no matter if I use v2s1 or v2s2. Resulting manifest.json contains digests for all blobs (different image but built at the same time with the same script and failed to get uploaded with the same error):</font></div><div><font face="monospace">{'config': {'digest': 'sha256:fef5b5f9bb6fca5f1909303822ae34ae9d3ac1d88da84144fa1a7b83229652f3',<br> 'mediaType': 'application/vnd.docker.container.image.v1+json',<br> 'size': 2339},<br> 'layers': [{'digest': 'sha256:b05580fca2f9aabb2d8fa975b29146c9147c8418e559f197c54a4fac04babb95',<br> 'mediaType': 'application/vnd.docker.image.rootfs.diff.tar.gzip',<br> 'size': 210204160},<br> {'digest': 'sha256:bbd20206b01e4bf5d521890bf5ffe2d5f565bbf70eda987b059b267c80f4f9a1',<br> 'mediaType': 'application/vnd.docker.image.rootfs.diff.tar.gzip',<br> 'size': 39936}],<br> 'mediaType': 'application/vnd.docker.distribution.manifest.v2+json',<br> 'schemaVersion': 2}</font><br></div><div></div></div><br><div class="gmail_quote"><div dir="ltr" class="gmail_attr">ср, 24 окт. 2018 г. в 16:05, Konstantin M. Khankin <<a href="mailto:khankin.konstantin@gmail.com" target="_blank">khankin.konstantin@gmail.com</a>>:<br></div><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex"><div dir="ltr"><div dir="ltr">Thank you, <a href="https://pulp.plan.io/issues/4095" target="_blank">https://pulp.plan.io/issues/4095</a> filed (why didn't I wait a bit to get 4096...)</div><br><div class="gmail_quote"><div dir="ltr">ср, 24 окт. 2018 г. в 15:35, Austin Macdonald <<a href="mailto:amacdona@redhat.com" target="_blank">amacdona@redhat.com</a>>:<br></div><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex"><div dir="ltr"><div dir="ltr">The other projects are sometimes hidden. <a href="https://pulp.plan.io/projects/pulp_docker" target="_blank">https://pulp.plan.io/projects/pulp_docker</a><br></div></div><br><div class="gmail_quote"><div dir="ltr">On Wed, Oct 24, 2018 at 2:55 AM Konstantin M. Khankin <<a href="mailto:khankin.konstantin@gmail.com" target="_blank">khankin.konstantin@gmail.com</a>> wrote:<br></div><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex"><div dir="ltr">Dennis,<div><br></div><div>Which project I should use? I don't see a separate project for pulp_docker.</div><div><br></div><div>Thanks!</div><br><div class="gmail_quote"><div dir="ltr">вт, 23 окт. 2018 г. в 23:58, Dennis Kliban <<a href="mailto:dkliban@redhat.com" target="_blank">dkliban@redhat.com</a>>:<br></div><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex"><div dir="ltr"><div>Konstantin, <br></div><div><br></div><div>Could you please file an issue in <a href="http://pulp.plan.io" target="_blank">pulp.plan.io</a>? <br></div><div><br></div><div>Thanks,</div><div>Dennis<br></div></div><br><div class="gmail_quote"><div dir="ltr">On Sat, Oct 20, 2018 at 5:40 AM Konstantin M. Khankin <<a href="mailto:khankin.konstantin@gmail.com" target="_blank">khankin.konstantin@gmail.com</a>> wrote:<br></div><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex"><div dir="ltr"><div dir="ltr"><div dir="ltr"><div dir="ltr"><div dir="ltr"><div dir="ltr"><b>Forgot to mention versions:</b><div><br></div><div><div>pulp-admin-client-2.17.1-1.el7.noarch</div><div>pulp-agent-2.17.1-1.el7.noarch</div><div>pulp-consumer-client-2.17.1-1.el7.noarch</div><div>pulp-deb-admin-extensions-1.8.0-1.el7.noarch</div><div>pulp-deb-plugins-1.8.0-1.el7.noarch</div><div>pulp-docker-admin-extensions-3.2.1-1.el7.noarch</div><div>pulp-docker-plugins-3.2.1-1.el7.noarch</div><div>pulp-puppet-consumer-extensions-2.17.1-1.el7.noarch</div><div>pulp-puppet-handlers-2.17.1-1.el7.noarch</div><div>pulp-python-admin-extensions-2.0.3-1.el7.noarch</div><div>pulp-python-plugins-2.0.3-1.el7.noarch</div><div>pulp-rpm-admin-extensions-2.17.1-1.el7.noarch</div><div>pulp-rpm-consumer-extensions-2.17.1-1.el7.noarch</div><div>pulp-rpm-handlers-2.17.1-1.el7.noarch</div><div>pulp-rpm-plugins-2.17.1-1.el7.noarch</div><div>pulp-rpm-yumplugins-2.17.1-1.el7.noarch</div><div>pulp-selinux-2.17.1-1.el7.noarch</div><div>pulp-server-2.17.1-1.el7.noarch</div><div>python-pulp-agent-lib-2.17.1-1.el7.noarch</div><div>python-pulp-bindings-2.17.1-1.el7.noarch</div><div>python-pulp-client-lib-2.17.1-1.el7.noarch</div><div>python-pulp-common-2.17.1-1.el7.noarch</div><div>python-pulp-deb-common-1.8.0-1.el7.noarch</div><div>python-pulp-docker-common-3.2.1-1.el7.noarch</div><div>python-pulp-oid_validation-2.17.1-1.el7.noarch</div><div>python-pulp-puppet-common-2.17.1-1.el7.noarch</div><div>python-pulp-python-common-2.0.3-1.el7.noarch</div><div>python-pulp-repoauth-2.17.1-1.el7.noarch</div><div>python-pulp-rpm-common-2.17.1-1.el7.noarch</div></div><div><br></div><div>buildah-1.2-2.gitbe87762.el7.x86_64<br></div><div><br></div><div><b>Manifest v2s1 from one of the images created by buildah:</b></div><div><div><font face="monospace, monospace">{'architecture': 'amd64',</font></div><div><font face="monospace, monospace"> 'fsLayers': [{'blobSum': 'sha256:4d85b6fa2344380743db8ccf4f0404603f56c8289db0824c13bfc309b2009be8'},</font></div><div><font face="monospace, monospace"> {'blobSum': 'sha256:a3ed95caeb02ffe68cdd9fd84406680ae93d633cb16422d00e8a7c22955b46d4'},</font></div><div><font face="monospace, monospace"> {'blobSum': 'sha256:a3ed95caeb02ffe68cdd9fd84406680ae93d633cb16422d00e8a7c22955b46d4'},</font></div><div><font face="monospace, monospace"> {'blobSum': 'sha256:f972d139738dfcd1519fd2461815651336ee25a8b54c358834c50af094bb262f'}],</font></div><div><font face="monospace, monospace"> 'history': [{'v1Compatibility': '{"architecture":"amd64","config":{"Env":["PATH=/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin"],"Entrypoint":["/bin/bash"],"WorkingDir":"/","Labels":{"org.label-schema.build-date":"20181006","org.label-schema.license":"GPLv2","<a href="http://org.label-schema.name" target="_blank">org.label-schema.name</a>":"CentOS Base Image","org.label-schema.schema-version":"1.0","org.label-schema.vendor":"CentOS"}},"created":"2018-10-20T07:49:56.084883454Z","id":"4591c91f4c223fa61c800a84bcf711647786f3fe71e589606f17b4d11b9c5807","os":"linux","parent":"f871460fb057cfa08e7268df9b884eecedeed72bc6dbc3262deb19a25e66a904"}'},</font></div><div><font face="monospace, monospace"> {'v1Compatibility': '{"id":"f871460fb057cfa08e7268df9b884eecedeed72bc6dbc3262deb19a25e66a904","parent":"8a4f409d848e6f693d5a4e6c260fbd8ec6a4dc99fc9915e3210bd5ed9c22e700","created":"2018-10-09T18:19:48.447478476Z","container_config":{"Cmd":["/bin/sh -c #(nop) CMD [\\"/bin/bash\\"]"]},"throwaway":true}'},</font></div><div><font face="monospace, monospace"> {'v1Compatibility': '{"id":"8a4f409d848e6f693d5a4e6c260fbd8ec6a4dc99fc9915e3210bd5ed9c22e700","parent":"961164371d3af72a151063f656820bc835c695fa44af66c2845d0850f01ce0d9","created":"2018-10-09T18:19:48.258784779Z","container_config":{"Cmd":["/bin/sh -c #(nop) LABEL org.label-schema.schema-version=1.0 <a href="http://org.label-schema.name" target="_blank">org.label-schema.name</a>=CentOS Base Image org.label-schema.vendor=CentOS org.label-schema.license=GPLv2 org.label-schema.build-date=20181006"]},"throwaway":true}'},</font></div><div><font face="monospace, monospace"> {'v1Compatibility': '{"id":"961164371d3af72a151063f656820bc835c695fa44af66c2845d0850f01ce0d9","created":"2018-10-09T18:19:47.884079751Z","container_config":{"Cmd":["/bin/sh -c #(nop) ADD file:fbe9badfd2790f0747a25fbe5c94a6daa78969511ca08c8d4ac654f3442570de in / "]}}'}],</font></div><div><font face="monospace, monospace"> 'name': '',</font></div><div><font face="monospace, monospace"> 'schemaVersion': 1,</font></div><div><font face="monospace, monospace"> 'signatures': [{'header': {'alg': 'ES256',</font></div><div><font face="monospace, monospace"> 'jwk': {'crv': 'P-256',</font></div><div><font face="monospace, monospace"> 'kid': '...',</font></div><div><font face="monospace, monospace"> 'kty': 'EC',</font></div><div><font face="monospace, monospace"> 'x': '...',</font></div><div><font face="monospace, monospace"> 'y': '...'}},</font></div><div><font face="monospace, monospace"> 'protected': '...',</font></div><div><font face="monospace, monospace"> 'signature': '...'}],</font></div><div><font face="monospace, monospace"> 'tag': ''}</font></div></div><div><font face="monospace, monospace"><br></font></div><div><font face="arial, helvetica, sans-serif"><b>Manifest v2s2 for the same image:</b></font></div><div><div><font face="monospace, monospace">{'config': {'digest': 'sha256:c17d63d23edcbe46b14fbe531e2e9b4411483947669087e90bdba3372eb2684d',</font></div><div style="font-family:monospace,monospace"> 'mediaType': 'application/vnd.docker.container.image.v1+json',</div><div style="font-family:monospace,monospace"> 'size': 1217},</div><div style="font-family:monospace,monospace"> 'layers': [{'digest': 'sha256:f972d139738dfcd1519fd2461815651336ee25a8b54c358834c50af094bb262f',</div><div style="font-family:monospace,monospace"> 'mediaType': 'application/vnd.docker.image.rootfs.diff.tar.gzip',</div><div style="font-family:monospace,monospace"> 'size': 208811520},</div><div style="font-family:monospace,monospace"> {'digest': 'sha256:4d85b6fa2344380743db8ccf4f0404603f56c8289db0824c13bfc309b2009be8',</div><div style="font-family:monospace,monospace"> 'mediaType': 'application/vnd.docker.image.rootfs.diff.tar.gzip',</div><div style="font-family:monospace,monospace"> 'size': 589285888}],</div><div style="font-family:monospace,monospace"> 'mediaType': 'application/vnd.docker.distribution.manifest.v2+json',</div><div style="font-family:monospace,monospace"> 'schemaVersion': 2}</div></div></div></div></div></div></div></div><br><div class="gmail_quote"><div dir="ltr">сб, 20 окт. 2018 г. в 11:56, Konstantin M. Khankin <<a href="mailto:khankin.konstantin@gmail.com" target="_blank">khankin.konstantin@gmail.com</a>>:<br></div><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex"><div dir="ltr"><div dir="ltr"><div dir="ltr"><div dir="ltr"><div dir="ltr"><div dir="ltr">Hi!<div><br></div><div>I tried the tool Buildah (<a href="https://www.projectatomic.io/blog/2017/06/introducing-buildah/" target="_blank">https://www.projectatomic.io/blog/2017/06/introducing-buildah/</a>) to build containers without having docker binary. Containers are built (not sure if they work), but I can't upload them to the pulp repo:</div><div><br></div><div>1) I copy container from "container-storage:" to "dir:", then create tar archive per <a href="https://docs.pulpproject.org/plugins/pulp_docker/user-guide/recipes.html#upload-v2-schema-2-and-schema-1-images-to-pulp" target="_blank">https://docs.pulpproject.org/plugins/pulp_docker/user-guide/recipes.html#upload-v2-schema-2-and-schema-1-images-to-pulp</a></div><div>2) pulp-admin -vvv docker repo uploads upload --repo-id docker_centos7 -f centos7.tar</div><div><br></div><div>[2] results in:</div><div><div> "traceback": "Traceback (most recent call last):\n File \"/usr/lib/python2.7/site-packages/celery/app/trace.py\", line 367, in trace_task\n R = retval = fun(*args, **kwargs)\n File \"/usr/lib/python2.7/site-packages/pulp/server/async/tasks.py\", line 529, in __call__\n return super(Task, self).__call__(*args, **kwargs)\n File \"/usr/lib/python2.7/site-packages/pulp/server/async/tasks.py\", line 107, in __call__\n return super(PulpTask, self).__call__(*args, **kwargs)\n File \"/usr/lib/python2.7/site-packages/celery/app/trace.py\", line 622, in __protected_call__\n return self.run(*args, **kwargs)\n File \"/usr/lib/python2.7/site-packages/pulp/server/managers/content/upload.py\", line 223, in import_uploaded_unit\n unit_type=unit_type_id, summary=result['summary'], details=result['details']\nPulpCodedException: The importer docker_importer indicated a failed response when uploading docker_manifest unit to repository docker_centos7.\n", </div></div><div><br></div><div><div> "error": {</div><div> "code": "PLP0047", </div><div> "data": {</div><div> "unit_type": "docker_manifest", </div><div> "importer_id": "docker_importer", </div><div> "repo_id": "docker_centos7", </div><div> "details": {}, </div><div> "summary": [</div><div> "Cannot resolve field \"digest\""</div><div> ]</div><div> }, </div><div> "description": "The importer docker_importer indicated a failed response when uploading docker_manifest unit to repository docker_centos7.", </div></div><div><div><br></div><div>I tried both v2s2 and v2s1, both don't work. I tried 2 different Dockerfiles and both produce the same error.</div><div><br></div><div>Is it an issue in Buildah or in Pulp? Or am I doing something wrong?</div><div><br></div><div>Thanks!</div><div><br></div>-- <br><div dir="ltr">Konstantin Khankin<br></div></div></div></div></div></div></div></div> </blockquote></div><br clear="all"><div><br></div>-- <br><div dir="ltr">Ханкин Константин<br></div> _______________________________________________<br> Pulp-list mailing list<br> <a href="mailto:Pulp-list@redhat.com" target="_blank">Pulp-list@redhat.com</a><br> <a href="https://www.redhat.com/mailman/listinfo/pulp-list" rel="noreferrer" target="_blank">https://www.redhat.com/mailman/listinfo/pulp-list</a></blockquote></div> </blockquote></div><br clear="all"><div><br></div>-- <br><div>Konstantin Khankin</div></div> _______________________________________________<br> Pulp-list mailing list<br> <a href="mailto:Pulp-list@redhat.com" target="_blank">Pulp-list@redhat.com</a><br> <a href="https://www.redhat.com/mailman/listinfo/pulp-list" rel="noreferrer" target="_blank">https://www.redhat.com/mailman/listinfo/pulp-list</a></blockquote></div> </blockquote></div><br clear="all"><div><br></div>-- <br><div dir="ltr">Konstantin Khankin<br></div></div> </blockquote></div><br clear="all"><div><br></div>-- <br><div dir="ltr" class="gmail-m_2779909613220660738gmail_signature">Ханкин Константин<br></div> _______________________________________________<br> Pulp-list mailing list<br> <a href="mailto:Pulp-list@redhat.com" target="_blank">Pulp-list@redhat.com</a><br> <a href="https://www.redhat.com/mailman/listinfo/pulp-list" rel="noreferrer" target="_blank">https://www.redhat.com/mailman/listinfo/pulp-list</a></blockquote></div>