[Pulp-list] Can't import containers created with Buildah
Ina Panova
ipanova at redhat.com
Wed Sep 25 16:23:58 UTC 2019
Hi Konstantin,
surprisingly today i was able to reproduce the issue and was able to find
the root cause of the problem.
Here's the PR that should fix the upload issue
https://github.com/pulp/pulp_docker/pull/418
--------
Regards,
Ina Panova
Senior Software Engineer| Pulp| Red Hat Inc.
"Do not go where the path may lead,
go instead where there is no path and leave a trail."
On Tue, Sep 17, 2019 at 11:58 AM Konstantin M. Khankin <
khankin.konstantin at gmail.com> wrote:
> Sorry for necroposting. I'm not sure if I still have the same issue, but I
> have the same symptoms. Could you please have a look and advice? Thanks!
>
> Whenever I try to create and upload another version of a container to the
> same repo, I get an error message like this:
>
> $ /usr/bin/pulp-admin docker repo uploads upload --repo-id gsk_postfix -f
> /tmp/postfix.tar
> ...
> Uploading: postfix.tar
> [==================================================] 100%
> 283176960/283176960 bytes
> ... completed
>
> Importing into the repository...
> This command may be exited via ctrl+c without affecting the request.
>
>
> [-]
> Running...
>
> Task Failed
>
> The importer docker_importer indicated a failed response when uploading
> docker_manifest unit to repository gsk_postfix. Summary: 'Cannot resolve
> field
> "digest"'. Details: '{}'
>
> I run pulp 2.20.1 and use buildah 1.9.0 to build container images. First
> image in the repository uploads fine, but the second always fails.
>
> Current repo status:
> Id: gsk_postfix
> Display Name: None
> Description: None
> Content Unit Counts:
> Docker Blob: 4
> Docker Manifest: 1
> Docker Tag: 1
>
> # pulp-admin docker repo search image --repo-id gsk_postfix
> <returns nothing>
>
> # pulp-admin docker repo search manifest --repo-id gsk_postfix
> Created: 2019-08-20T22:58:50Z
> Metadata:
> Config Layer:
> sha256:d2b497c83fe099624567046432468d84a5fdb49139b5192c9b332f0b10784695
> Digest:
> sha256:d17067c71cb229f703b0f43ecd8a85c55995f3007b99570f580f9f6ebc5dbccc
> Downloaded: True
> Fs Layers:
> Blob Sum:
> sha256:b05580fca2f9aabb2d8fa975b29146c9147c8418e559f197c54a4fac04babb95
> Layer Type: application/vnd.docker.image.rootfs.diff.tar.gzip
> Size: 210204160
> Blob Sum:
> sha256:204cafbfdb1d31cb427478102377a99ceccfd7cc7be19cc191568fb9008fd4af
> Layer Type: application/vnd.docker.image.rootfs.diff.tar.gzip
> Size: 12288
> Blob Sum:
> sha256:a85e1551e989545ca081c9ed0584e2db535a1484b91060eed88a17991e4b25cf
> Layer Type: application/vnd.docker.image.rootfs.diff.tar.gzip
> Size: 72946688
> Pulp User Metadata:
> Schema Version: 2
> Repo Id: gsk_postfix
> Unit Id: 000c0b4a-fb98-4ca3-a3ec-3b2a42521949
> Unit Type Id: docker_manifest
> Updated: 2019-08-20T22:58:50Z
>
> # pulp-admin docker repo search tag --repo-id gsk_postfix
> Created: 2019-08-20T22:58:55Z
> Metadata:
> Manifest Digest:
> sha256:d17067c71cb229f703b0f43ecd8a85c55995f3007b99570f580f9f6ebc5dbccc
> Manifest Type: image
> Name: latest
> Pulp User Metadata:
> Repo Id: gsk_postfix
> Schema Version: 2
> Repo Id: gsk_postfix
> Unit Id: 26c40bb1-4f3b-438f-8f50-cc5afe6baeb6
> Unit Type Id: docker_tag
> Updated: 2019-08-20T22:58:55Z
>
> To build and upload containers, I wrote a simple script. Here is the
> command sequence:
> ### STEP 1: Build an image from Dockerfile.
> $ /usr/bin/buildah bud -t gsk_postfix .
> ...
> Writing manifest to image destination
> Storing signatures
> db008cf80da01afe8ca11898e566279a7765e682bf56070857d40ffd469e2486
> ### STEP 2: Get the image digest.
> $ /usr/bin/buildah images --quiet --digests --no-trunc
> localhost/gsk_postfix:latest
> sha256:db008cf80da01afe8ca11898e566279a7765e682bf56070857d40ffd469e2486
> ### STEP 3: Pull blobs from the storage.
> $ /usr/bin/skopeo copy --format v2s2
> containers-storage:localhost/gsk_postfix:latest dir:/tmp/tmp.NYhxiiOsOe
> Getting image source signatures
> Copying blob b05580fca2f9 done
> Copying blob 8d136bb340b2 done
> Copying blob 7a406e7184ab done
> Copying config db008cf80d done
> Writing manifest to image destination
> Storing signatures
> ### STEP 4: Tar the archive.
> $ tar cf /tmp/postfix.tar *
> ### STEP 5: Upload the archive to the repo.
> $ /usr/bin/pulp-admin docker repo uploads upload --repo-id gsk_postfix -f
> /tmp/postfix.tar
> <failure here>
>
> This happens no matter if I use v2s1 or v2s2. Resulting manifest.json
> contains digests for all blobs (different image but built at the same time
> with the same script and failed to get uploaded with the same error):
> {'config': {'digest':
> 'sha256:fef5b5f9bb6fca5f1909303822ae34ae9d3ac1d88da84144fa1a7b83229652f3',
> 'mediaType': 'application/vnd.docker.container.image.v1+json',
> 'size': 2339},
> 'layers': [{'digest':
> 'sha256:b05580fca2f9aabb2d8fa975b29146c9147c8418e559f197c54a4fac04babb95',
> 'mediaType':
> 'application/vnd.docker.image.rootfs.diff.tar.gzip',
> 'size': 210204160},
> {'digest':
> 'sha256:bbd20206b01e4bf5d521890bf5ffe2d5f565bbf70eda987b059b267c80f4f9a1',
> 'mediaType':
> 'application/vnd.docker.image.rootfs.diff.tar.gzip',
> 'size': 39936}],
> 'mediaType': 'application/vnd.docker.distribution.manifest.v2+json',
> 'schemaVersion': 2}
>
> ср, 24 окт. 2018 г. в 16:05, Konstantin M. Khankin <
> khankin.konstantin at gmail.com>:
>
>> Thank you, https://pulp.plan.io/issues/4095 filed (why didn't I wait a
>> bit to get 4096...)
>>
>> ср, 24 окт. 2018 г. в 15:35, Austin Macdonald <amacdona at redhat.com>:
>>
>>> The other projects are sometimes hidden.
>>> https://pulp.plan.io/projects/pulp_docker
>>>
>>> On Wed, Oct 24, 2018 at 2:55 AM Konstantin M. Khankin <
>>> khankin.konstantin at gmail.com> wrote:
>>>
>>>> Dennis,
>>>>
>>>> Which project I should use? I don't see a separate project for
>>>> pulp_docker.
>>>>
>>>> Thanks!
>>>>
>>>> вт, 23 окт. 2018 г. в 23:58, Dennis Kliban <dkliban at redhat.com>:
>>>>
>>>>> Konstantin,
>>>>>
>>>>> Could you please file an issue in pulp.plan.io?
>>>>>
>>>>> Thanks,
>>>>> Dennis
>>>>>
>>>>> On Sat, Oct 20, 2018 at 5:40 AM Konstantin M. Khankin <
>>>>> khankin.konstantin at gmail.com> wrote:
>>>>>
>>>>>> *Forgot to mention versions:*
>>>>>>
>>>>>> pulp-admin-client-2.17.1-1.el7.noarch
>>>>>> pulp-agent-2.17.1-1.el7.noarch
>>>>>> pulp-consumer-client-2.17.1-1.el7.noarch
>>>>>> pulp-deb-admin-extensions-1.8.0-1.el7.noarch
>>>>>> pulp-deb-plugins-1.8.0-1.el7.noarch
>>>>>> pulp-docker-admin-extensions-3.2.1-1.el7.noarch
>>>>>> pulp-docker-plugins-3.2.1-1.el7.noarch
>>>>>> pulp-puppet-consumer-extensions-2.17.1-1.el7.noarch
>>>>>> pulp-puppet-handlers-2.17.1-1.el7.noarch
>>>>>> pulp-python-admin-extensions-2.0.3-1.el7.noarch
>>>>>> pulp-python-plugins-2.0.3-1.el7.noarch
>>>>>> pulp-rpm-admin-extensions-2.17.1-1.el7.noarch
>>>>>> pulp-rpm-consumer-extensions-2.17.1-1.el7.noarch
>>>>>> pulp-rpm-handlers-2.17.1-1.el7.noarch
>>>>>> pulp-rpm-plugins-2.17.1-1.el7.noarch
>>>>>> pulp-rpm-yumplugins-2.17.1-1.el7.noarch
>>>>>> pulp-selinux-2.17.1-1.el7.noarch
>>>>>> pulp-server-2.17.1-1.el7.noarch
>>>>>> python-pulp-agent-lib-2.17.1-1.el7.noarch
>>>>>> python-pulp-bindings-2.17.1-1.el7.noarch
>>>>>> python-pulp-client-lib-2.17.1-1.el7.noarch
>>>>>> python-pulp-common-2.17.1-1.el7.noarch
>>>>>> python-pulp-deb-common-1.8.0-1.el7.noarch
>>>>>> python-pulp-docker-common-3.2.1-1.el7.noarch
>>>>>> python-pulp-oid_validation-2.17.1-1.el7.noarch
>>>>>> python-pulp-puppet-common-2.17.1-1.el7.noarch
>>>>>> python-pulp-python-common-2.0.3-1.el7.noarch
>>>>>> python-pulp-repoauth-2.17.1-1.el7.noarch
>>>>>> python-pulp-rpm-common-2.17.1-1.el7.noarch
>>>>>>
>>>>>> buildah-1.2-2.gitbe87762.el7.x86_64
>>>>>>
>>>>>> *Manifest v2s1 from one of the images created by buildah:*
>>>>>> {'architecture': 'amd64',
>>>>>> 'fsLayers': [{'blobSum':
>>>>>> 'sha256:4d85b6fa2344380743db8ccf4f0404603f56c8289db0824c13bfc309b2009be8'},
>>>>>> {'blobSum':
>>>>>> 'sha256:a3ed95caeb02ffe68cdd9fd84406680ae93d633cb16422d00e8a7c22955b46d4'},
>>>>>> {'blobSum':
>>>>>> 'sha256:a3ed95caeb02ffe68cdd9fd84406680ae93d633cb16422d00e8a7c22955b46d4'},
>>>>>> {'blobSum':
>>>>>> 'sha256:f972d139738dfcd1519fd2461815651336ee25a8b54c358834c50af094bb262f'}],
>>>>>> 'history': [{'v1Compatibility':
>>>>>> '{"architecture":"amd64","config":{"Env":["PATH=/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin"],"Entrypoint":["/bin/bash"],"WorkingDir":"/","Labels":{"org.label-schema.build-date":"20181006","org.label-schema.license":"GPLv2","
>>>>>> org.label-schema.name":"CentOS Base
>>>>>> Image","org.label-schema.schema-version":"1.0","org.label-schema.vendor":"CentOS"}},"created":"2018-10-20T07:49:56.084883454Z","id":"4591c91f4c223fa61c800a84bcf711647786f3fe71e589606f17b4d11b9c5807","os":"linux","parent":"f871460fb057cfa08e7268df9b884eecedeed72bc6dbc3262deb19a25e66a904"}'},
>>>>>> {'v1Compatibility':
>>>>>> '{"id":"f871460fb057cfa08e7268df9b884eecedeed72bc6dbc3262deb19a25e66a904","parent":"8a4f409d848e6f693d5a4e6c260fbd8ec6a4dc99fc9915e3210bd5ed9c22e700","created":"2018-10-09T18:19:48.447478476Z","container_config":{"Cmd":["/bin/sh
>>>>>> -c #(nop) CMD [\\"/bin/bash\\"]"]},"throwaway":true}'},
>>>>>> {'v1Compatibility':
>>>>>> '{"id":"8a4f409d848e6f693d5a4e6c260fbd8ec6a4dc99fc9915e3210bd5ed9c22e700","parent":"961164371d3af72a151063f656820bc835c695fa44af66c2845d0850f01ce0d9","created":"2018-10-09T18:19:48.258784779Z","container_config":{"Cmd":["/bin/sh
>>>>>> -c #(nop) LABEL org.label-schema.schema-version=1.0
>>>>>> org.label-schema.name=CentOS Base Image
>>>>>> org.label-schema.vendor=CentOS org.label-schema.license=GPLv2
>>>>>> org.label-schema.build-date=20181006"]},"throwaway":true}'},
>>>>>> {'v1Compatibility':
>>>>>> '{"id":"961164371d3af72a151063f656820bc835c695fa44af66c2845d0850f01ce0d9","created":"2018-10-09T18:19:47.884079751Z","container_config":{"Cmd":["/bin/sh
>>>>>> -c #(nop) ADD
>>>>>> file:fbe9badfd2790f0747a25fbe5c94a6daa78969511ca08c8d4ac654f3442570de in /
>>>>>> "]}}'}],
>>>>>> 'name': '',
>>>>>> 'schemaVersion': 1,
>>>>>> 'signatures': [{'header': {'alg': 'ES256',
>>>>>> 'jwk': {'crv': 'P-256',
>>>>>> 'kid': '...',
>>>>>> 'kty': 'EC',
>>>>>> 'x': '...',
>>>>>> 'y': '...'}},
>>>>>> 'protected': '...',
>>>>>> 'signature': '...'}],
>>>>>> 'tag': ''}
>>>>>>
>>>>>> *Manifest v2s2 for the same image:*
>>>>>> {'config': {'digest':
>>>>>> 'sha256:c17d63d23edcbe46b14fbe531e2e9b4411483947669087e90bdba3372eb2684d',
>>>>>> 'mediaType':
>>>>>> 'application/vnd.docker.container.image.v1+json',
>>>>>> 'size': 1217},
>>>>>> 'layers': [{'digest':
>>>>>> 'sha256:f972d139738dfcd1519fd2461815651336ee25a8b54c358834c50af094bb262f',
>>>>>> 'mediaType':
>>>>>> 'application/vnd.docker.image.rootfs.diff.tar.gzip',
>>>>>> 'size': 208811520},
>>>>>> {'digest':
>>>>>> 'sha256:4d85b6fa2344380743db8ccf4f0404603f56c8289db0824c13bfc309b2009be8',
>>>>>> 'mediaType':
>>>>>> 'application/vnd.docker.image.rootfs.diff.tar.gzip',
>>>>>> 'size': 589285888}],
>>>>>> 'mediaType': 'application/vnd.docker.distribution.manifest.v2+json',
>>>>>> 'schemaVersion': 2}
>>>>>>
>>>>>> сб, 20 окт. 2018 г. в 11:56, Konstantin M. Khankin <
>>>>>> khankin.konstantin at gmail.com>:
>>>>>>
>>>>>>> Hi!
>>>>>>>
>>>>>>> I tried the tool Buildah (
>>>>>>> https://www.projectatomic.io/blog/2017/06/introducing-buildah/) to
>>>>>>> build containers without having docker binary. Containers are built (not
>>>>>>> sure if they work), but I can't upload them to the pulp repo:
>>>>>>>
>>>>>>> 1) I copy container from "container-storage:" to "dir:", then create
>>>>>>> tar archive per
>>>>>>> https://docs.pulpproject.org/plugins/pulp_docker/user-guide/recipes.html#upload-v2-schema-2-and-schema-1-images-to-pulp
>>>>>>> 2) pulp-admin -vvv docker repo uploads upload --repo-id
>>>>>>> docker_centos7 -f centos7.tar
>>>>>>>
>>>>>>> [2] results in:
>>>>>>> "traceback": "Traceback (most recent call last):\n File
>>>>>>> \"/usr/lib/python2.7/site-packages/celery/app/trace.py\", line 367, in
>>>>>>> trace_task\n R = retval = fun(*args, **kwargs)\n File
>>>>>>> \"/usr/lib/python2.7/site-packages/pulp/server/async/tasks.py\", line 529,
>>>>>>> in __call__\n return super(Task, self).__call__(*args, **kwargs)\n File
>>>>>>> \"/usr/lib/python2.7/site-packages/pulp/server/async/tasks.py\", line 107,
>>>>>>> in __call__\n return super(PulpTask, self).__call__(*args, **kwargs)\n
>>>>>>> File \"/usr/lib/python2.7/site-packages/celery/app/trace.py\", line 622, in
>>>>>>> __protected_call__\n return self.run(*args, **kwargs)\n File
>>>>>>> \"/usr/lib/python2.7/site-packages/pulp/server/managers/content/upload.py\",
>>>>>>> line 223, in import_uploaded_unit\n unit_type=unit_type_id,
>>>>>>> summary=result['summary'], details=result['details']\nPulpCodedException:
>>>>>>> The importer docker_importer indicated a failed response when uploading
>>>>>>> docker_manifest unit to repository docker_centos7.\n",
>>>>>>>
>>>>>>> "error": {
>>>>>>> "code": "PLP0047",
>>>>>>> "data": {
>>>>>>> "unit_type": "docker_manifest",
>>>>>>> "importer_id": "docker_importer",
>>>>>>> "repo_id": "docker_centos7",
>>>>>>> "details": {},
>>>>>>> "summary": [
>>>>>>> "Cannot resolve field \"digest\""
>>>>>>> ]
>>>>>>> },
>>>>>>> "description": "The importer docker_importer indicated a failed
>>>>>>> response when uploading docker_manifest unit to repository
>>>>>>> docker_centos7.",
>>>>>>>
>>>>>>> I tried both v2s2 and v2s1, both don't work. I tried 2 different
>>>>>>> Dockerfiles and both produce the same error.
>>>>>>>
>>>>>>> Is it an issue in Buildah or in Pulp? Or am I doing something wrong?
>>>>>>>
>>>>>>> Thanks!
>>>>>>>
>>>>>>> --
>>>>>>> Konstantin Khankin
>>>>>>>
>>>>>>
>>>>>>
>>>>>> --
>>>>>> Ханкин Константин
>>>>>> _______________________________________________
>>>>>> Pulp-list mailing list
>>>>>> Pulp-list at redhat.com
>>>>>> https://www.redhat.com/mailman/listinfo/pulp-list
>>>>>
>>>>>
>>>>
>>>> --
>>>> Konstantin Khankin
>>>> _______________________________________________
>>>> Pulp-list mailing list
>>>> Pulp-list at redhat.com
>>>> https://www.redhat.com/mailman/listinfo/pulp-list
>>>
>>>
>>
>> --
>> Konstantin Khankin
>>
>
>
> --
> Ханкин Константин
> _______________________________________________
> Pulp-list mailing list
> Pulp-list at redhat.com
> https://www.redhat.com/mailman/listinfo/pulp-list
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://listman.redhat.com/archives/pulp-list/attachments/20190925/152256ee/attachment.htm>
More information about the Pulp-list
mailing list