[Pulp-list] Can't import containers created with Buildah
Konstantin M. Khankin
khankin.konstantin at gmail.com
Thu Sep 26 12:11:23 UTC 2019
Thank you Ina!
I manually applied this patch and looks like it actually works :)
ср, 25 сент. 2019 г. в 19:24, Ina Panova <ipanova at redhat.com>:
> Hi Konstantin,
>
> surprisingly today i was able to reproduce the issue and was able to find
> the root cause of the problem.
> Here's the PR that should fix the upload issue
> https://github.com/pulp/pulp_docker/pull/418
>
>
> --------
> Regards,
>
> Ina Panova
> Senior Software Engineer| Pulp| Red Hat Inc.
>
> "Do not go where the path may lead,
> go instead where there is no path and leave a trail."
>
>
> On Tue, Sep 17, 2019 at 11:58 AM Konstantin M. Khankin <
> khankin.konstantin at gmail.com> wrote:
>
>> Sorry for necroposting. I'm not sure if I still have the same issue, but
>> I have the same symptoms. Could you please have a look and advice? Thanks!
>>
>> Whenever I try to create and upload another version of a container to the
>> same repo, I get an error message like this:
>>
>> $ /usr/bin/pulp-admin docker repo uploads upload --repo-id gsk_postfix -f
>> /tmp/postfix.tar
>> ...
>> Uploading: postfix.tar
>> [==================================================] 100%
>> 283176960/283176960 bytes
>> ... completed
>>
>> Importing into the repository...
>> This command may be exited via ctrl+c without affecting the request.
>>
>>
>> [-]
>> Running...
>>
>> Task Failed
>>
>> The importer docker_importer indicated a failed response when uploading
>> docker_manifest unit to repository gsk_postfix. Summary: 'Cannot resolve
>> field
>> "digest"'. Details: '{}'
>>
>> I run pulp 2.20.1 and use buildah 1.9.0 to build container images. First
>> image in the repository uploads fine, but the second always fails.
>>
>> Current repo status:
>> Id: gsk_postfix
>> Display Name: None
>> Description: None
>> Content Unit Counts:
>> Docker Blob: 4
>> Docker Manifest: 1
>> Docker Tag: 1
>>
>> # pulp-admin docker repo search image --repo-id gsk_postfix
>> <returns nothing>
>>
>> # pulp-admin docker repo search manifest --repo-id gsk_postfix
>> Created: 2019-08-20T22:58:50Z
>> Metadata:
>> Config Layer:
>> sha256:d2b497c83fe099624567046432468d84a5fdb49139b5192c9b332f0b10784695
>> Digest:
>> sha256:d17067c71cb229f703b0f43ecd8a85c55995f3007b99570f580f9f6ebc5dbccc
>> Downloaded: True
>> Fs Layers:
>> Blob Sum:
>> sha256:b05580fca2f9aabb2d8fa975b29146c9147c8418e559f197c54a4fac04babb95
>> Layer Type: application/vnd.docker.image.rootfs.diff.tar.gzip
>> Size: 210204160
>> Blob Sum:
>> sha256:204cafbfdb1d31cb427478102377a99ceccfd7cc7be19cc191568fb9008fd4af
>> Layer Type: application/vnd.docker.image.rootfs.diff.tar.gzip
>> Size: 12288
>> Blob Sum:
>> sha256:a85e1551e989545ca081c9ed0584e2db535a1484b91060eed88a17991e4b25cf
>> Layer Type: application/vnd.docker.image.rootfs.diff.tar.gzip
>> Size: 72946688
>> Pulp User Metadata:
>> Schema Version: 2
>> Repo Id: gsk_postfix
>> Unit Id: 000c0b4a-fb98-4ca3-a3ec-3b2a42521949
>> Unit Type Id: docker_manifest
>> Updated: 2019-08-20T22:58:50Z
>>
>> # pulp-admin docker repo search tag --repo-id gsk_postfix
>> Created: 2019-08-20T22:58:55Z
>> Metadata:
>> Manifest Digest:
>> sha256:d17067c71cb229f703b0f43ecd8a85c55995f3007b99570f580f9f6ebc5dbccc
>> Manifest Type: image
>> Name: latest
>> Pulp User Metadata:
>> Repo Id: gsk_postfix
>> Schema Version: 2
>> Repo Id: gsk_postfix
>> Unit Id: 26c40bb1-4f3b-438f-8f50-cc5afe6baeb6
>> Unit Type Id: docker_tag
>> Updated: 2019-08-20T22:58:55Z
>>
>> To build and upload containers, I wrote a simple script. Here is the
>> command sequence:
>> ### STEP 1: Build an image from Dockerfile.
>> $ /usr/bin/buildah bud -t gsk_postfix .
>> ...
>> Writing manifest to image destination
>> Storing signatures
>> db008cf80da01afe8ca11898e566279a7765e682bf56070857d40ffd469e2486
>> ### STEP 2: Get the image digest.
>> $ /usr/bin/buildah images --quiet --digests --no-trunc
>> localhost/gsk_postfix:latest
>> sha256:db008cf80da01afe8ca11898e566279a7765e682bf56070857d40ffd469e2486
>> ### STEP 3: Pull blobs from the storage.
>> $ /usr/bin/skopeo copy --format v2s2
>> containers-storage:localhost/gsk_postfix:latest dir:/tmp/tmp.NYhxiiOsOe
>> Getting image source signatures
>> Copying blob b05580fca2f9 done
>> Copying blob 8d136bb340b2 done
>> Copying blob 7a406e7184ab done
>> Copying config db008cf80d done
>> Writing manifest to image destination
>> Storing signatures
>> ### STEP 4: Tar the archive.
>> $ tar cf /tmp/postfix.tar *
>> ### STEP 5: Upload the archive to the repo.
>> $ /usr/bin/pulp-admin docker repo uploads upload --repo-id gsk_postfix -f
>> /tmp/postfix.tar
>> <failure here>
>>
>> This happens no matter if I use v2s1 or v2s2. Resulting manifest.json
>> contains digests for all blobs (different image but built at the same time
>> with the same script and failed to get uploaded with the same error):
>> {'config': {'digest':
>> 'sha256:fef5b5f9bb6fca5f1909303822ae34ae9d3ac1d88da84144fa1a7b83229652f3',
>> 'mediaType': 'application/vnd.docker.container.image.v1+json',
>> 'size': 2339},
>> 'layers': [{'digest':
>> 'sha256:b05580fca2f9aabb2d8fa975b29146c9147c8418e559f197c54a4fac04babb95',
>> 'mediaType':
>> 'application/vnd.docker.image.rootfs.diff.tar.gzip',
>> 'size': 210204160},
>> {'digest':
>> 'sha256:bbd20206b01e4bf5d521890bf5ffe2d5f565bbf70eda987b059b267c80f4f9a1',
>> 'mediaType':
>> 'application/vnd.docker.image.rootfs.diff.tar.gzip',
>> 'size': 39936}],
>> 'mediaType': 'application/vnd.docker.distribution.manifest.v2+json',
>> 'schemaVersion': 2}
>>
>> ср, 24 окт. 2018 г. в 16:05, Konstantin M. Khankin <
>> khankin.konstantin at gmail.com>:
>>
>>> Thank you, https://pulp.plan.io/issues/4095 filed (why didn't I wait a
>>> bit to get 4096...)
>>>
>>> ср, 24 окт. 2018 г. в 15:35, Austin Macdonald <amacdona at redhat.com>:
>>>
>>>> The other projects are sometimes hidden.
>>>> https://pulp.plan.io/projects/pulp_docker
>>>>
>>>> On Wed, Oct 24, 2018 at 2:55 AM Konstantin M. Khankin <
>>>> khankin.konstantin at gmail.com> wrote:
>>>>
>>>>> Dennis,
>>>>>
>>>>> Which project I should use? I don't see a separate project for
>>>>> pulp_docker.
>>>>>
>>>>> Thanks!
>>>>>
>>>>> вт, 23 окт. 2018 г. в 23:58, Dennis Kliban <dkliban at redhat.com>:
>>>>>
>>>>>> Konstantin,
>>>>>>
>>>>>> Could you please file an issue in pulp.plan.io?
>>>>>>
>>>>>> Thanks,
>>>>>> Dennis
>>>>>>
>>>>>> On Sat, Oct 20, 2018 at 5:40 AM Konstantin M. Khankin <
>>>>>> khankin.konstantin at gmail.com> wrote:
>>>>>>
>>>>>>> *Forgot to mention versions:*
>>>>>>>
>>>>>>> pulp-admin-client-2.17.1-1.el7.noarch
>>>>>>> pulp-agent-2.17.1-1.el7.noarch
>>>>>>> pulp-consumer-client-2.17.1-1.el7.noarch
>>>>>>> pulp-deb-admin-extensions-1.8.0-1.el7.noarch
>>>>>>> pulp-deb-plugins-1.8.0-1.el7.noarch
>>>>>>> pulp-docker-admin-extensions-3.2.1-1.el7.noarch
>>>>>>> pulp-docker-plugins-3.2.1-1.el7.noarch
>>>>>>> pulp-puppet-consumer-extensions-2.17.1-1.el7.noarch
>>>>>>> pulp-puppet-handlers-2.17.1-1.el7.noarch
>>>>>>> pulp-python-admin-extensions-2.0.3-1.el7.noarch
>>>>>>> pulp-python-plugins-2.0.3-1.el7.noarch
>>>>>>> pulp-rpm-admin-extensions-2.17.1-1.el7.noarch
>>>>>>> pulp-rpm-consumer-extensions-2.17.1-1.el7.noarch
>>>>>>> pulp-rpm-handlers-2.17.1-1.el7.noarch
>>>>>>> pulp-rpm-plugins-2.17.1-1.el7.noarch
>>>>>>> pulp-rpm-yumplugins-2.17.1-1.el7.noarch
>>>>>>> pulp-selinux-2.17.1-1.el7.noarch
>>>>>>> pulp-server-2.17.1-1.el7.noarch
>>>>>>> python-pulp-agent-lib-2.17.1-1.el7.noarch
>>>>>>> python-pulp-bindings-2.17.1-1.el7.noarch
>>>>>>> python-pulp-client-lib-2.17.1-1.el7.noarch
>>>>>>> python-pulp-common-2.17.1-1.el7.noarch
>>>>>>> python-pulp-deb-common-1.8.0-1.el7.noarch
>>>>>>> python-pulp-docker-common-3.2.1-1.el7.noarch
>>>>>>> python-pulp-oid_validation-2.17.1-1.el7.noarch
>>>>>>> python-pulp-puppet-common-2.17.1-1.el7.noarch
>>>>>>> python-pulp-python-common-2.0.3-1.el7.noarch
>>>>>>> python-pulp-repoauth-2.17.1-1.el7.noarch
>>>>>>> python-pulp-rpm-common-2.17.1-1.el7.noarch
>>>>>>>
>>>>>>> buildah-1.2-2.gitbe87762.el7.x86_64
>>>>>>>
>>>>>>> *Manifest v2s1 from one of the images created by buildah:*
>>>>>>> {'architecture': 'amd64',
>>>>>>> 'fsLayers': [{'blobSum':
>>>>>>> 'sha256:4d85b6fa2344380743db8ccf4f0404603f56c8289db0824c13bfc309b2009be8'},
>>>>>>> {'blobSum':
>>>>>>> 'sha256:a3ed95caeb02ffe68cdd9fd84406680ae93d633cb16422d00e8a7c22955b46d4'},
>>>>>>> {'blobSum':
>>>>>>> 'sha256:a3ed95caeb02ffe68cdd9fd84406680ae93d633cb16422d00e8a7c22955b46d4'},
>>>>>>> {'blobSum':
>>>>>>> 'sha256:f972d139738dfcd1519fd2461815651336ee25a8b54c358834c50af094bb262f'}],
>>>>>>> 'history': [{'v1Compatibility':
>>>>>>> '{"architecture":"amd64","config":{"Env":["PATH=/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin"],"Entrypoint":["/bin/bash"],"WorkingDir":"/","Labels":{"org.label-schema.build-date":"20181006","org.label-schema.license":"GPLv2","
>>>>>>> org.label-schema.name":"CentOS Base
>>>>>>> Image","org.label-schema.schema-version":"1.0","org.label-schema.vendor":"CentOS"}},"created":"2018-10-20T07:49:56.084883454Z","id":"4591c91f4c223fa61c800a84bcf711647786f3fe71e589606f17b4d11b9c5807","os":"linux","parent":"f871460fb057cfa08e7268df9b884eecedeed72bc6dbc3262deb19a25e66a904"}'},
>>>>>>> {'v1Compatibility':
>>>>>>> '{"id":"f871460fb057cfa08e7268df9b884eecedeed72bc6dbc3262deb19a25e66a904","parent":"8a4f409d848e6f693d5a4e6c260fbd8ec6a4dc99fc9915e3210bd5ed9c22e700","created":"2018-10-09T18:19:48.447478476Z","container_config":{"Cmd":["/bin/sh
>>>>>>> -c #(nop) CMD [\\"/bin/bash\\"]"]},"throwaway":true}'},
>>>>>>> {'v1Compatibility':
>>>>>>> '{"id":"8a4f409d848e6f693d5a4e6c260fbd8ec6a4dc99fc9915e3210bd5ed9c22e700","parent":"961164371d3af72a151063f656820bc835c695fa44af66c2845d0850f01ce0d9","created":"2018-10-09T18:19:48.258784779Z","container_config":{"Cmd":["/bin/sh
>>>>>>> -c #(nop) LABEL org.label-schema.schema-version=1.0
>>>>>>> org.label-schema.name=CentOS Base Image
>>>>>>> org.label-schema.vendor=CentOS org.label-schema.license=GPLv2
>>>>>>> org.label-schema.build-date=20181006"]},"throwaway":true}'},
>>>>>>> {'v1Compatibility':
>>>>>>> '{"id":"961164371d3af72a151063f656820bc835c695fa44af66c2845d0850f01ce0d9","created":"2018-10-09T18:19:47.884079751Z","container_config":{"Cmd":["/bin/sh
>>>>>>> -c #(nop) ADD
>>>>>>> file:fbe9badfd2790f0747a25fbe5c94a6daa78969511ca08c8d4ac654f3442570de in /
>>>>>>> "]}}'}],
>>>>>>> 'name': '',
>>>>>>> 'schemaVersion': 1,
>>>>>>> 'signatures': [{'header': {'alg': 'ES256',
>>>>>>> 'jwk': {'crv': 'P-256',
>>>>>>> 'kid': '...',
>>>>>>> 'kty': 'EC',
>>>>>>> 'x': '...',
>>>>>>> 'y': '...'}},
>>>>>>> 'protected': '...',
>>>>>>> 'signature': '...'}],
>>>>>>> 'tag': ''}
>>>>>>>
>>>>>>> *Manifest v2s2 for the same image:*
>>>>>>> {'config': {'digest':
>>>>>>> 'sha256:c17d63d23edcbe46b14fbe531e2e9b4411483947669087e90bdba3372eb2684d',
>>>>>>> 'mediaType':
>>>>>>> 'application/vnd.docker.container.image.v1+json',
>>>>>>> 'size': 1217},
>>>>>>> 'layers': [{'digest':
>>>>>>> 'sha256:f972d139738dfcd1519fd2461815651336ee25a8b54c358834c50af094bb262f',
>>>>>>> 'mediaType':
>>>>>>> 'application/vnd.docker.image.rootfs.diff.tar.gzip',
>>>>>>> 'size': 208811520},
>>>>>>> {'digest':
>>>>>>> 'sha256:4d85b6fa2344380743db8ccf4f0404603f56c8289db0824c13bfc309b2009be8',
>>>>>>> 'mediaType':
>>>>>>> 'application/vnd.docker.image.rootfs.diff.tar.gzip',
>>>>>>> 'size': 589285888}],
>>>>>>> 'mediaType': 'application/vnd.docker.distribution.manifest.v2+json',
>>>>>>> 'schemaVersion': 2}
>>>>>>>
>>>>>>> сб, 20 окт. 2018 г. в 11:56, Konstantin M. Khankin <
>>>>>>> khankin.konstantin at gmail.com>:
>>>>>>>
>>>>>>>> Hi!
>>>>>>>>
>>>>>>>> I tried the tool Buildah (
>>>>>>>> https://www.projectatomic.io/blog/2017/06/introducing-buildah/) to
>>>>>>>> build containers without having docker binary. Containers are built (not
>>>>>>>> sure if they work), but I can't upload them to the pulp repo:
>>>>>>>>
>>>>>>>> 1) I copy container from "container-storage:" to "dir:", then
>>>>>>>> create tar archive per
>>>>>>>> https://docs.pulpproject.org/plugins/pulp_docker/user-guide/recipes.html#upload-v2-schema-2-and-schema-1-images-to-pulp
>>>>>>>> 2) pulp-admin -vvv docker repo uploads upload --repo-id
>>>>>>>> docker_centos7 -f centos7.tar
>>>>>>>>
>>>>>>>> [2] results in:
>>>>>>>> "traceback": "Traceback (most recent call last):\n File
>>>>>>>> \"/usr/lib/python2.7/site-packages/celery/app/trace.py\", line 367, in
>>>>>>>> trace_task\n R = retval = fun(*args, **kwargs)\n File
>>>>>>>> \"/usr/lib/python2.7/site-packages/pulp/server/async/tasks.py\", line 529,
>>>>>>>> in __call__\n return super(Task, self).__call__(*args, **kwargs)\n File
>>>>>>>> \"/usr/lib/python2.7/site-packages/pulp/server/async/tasks.py\", line 107,
>>>>>>>> in __call__\n return super(PulpTask, self).__call__(*args, **kwargs)\n
>>>>>>>> File \"/usr/lib/python2.7/site-packages/celery/app/trace.py\", line 622, in
>>>>>>>> __protected_call__\n return self.run(*args, **kwargs)\n File
>>>>>>>> \"/usr/lib/python2.7/site-packages/pulp/server/managers/content/upload.py\",
>>>>>>>> line 223, in import_uploaded_unit\n unit_type=unit_type_id,
>>>>>>>> summary=result['summary'], details=result['details']\nPulpCodedException:
>>>>>>>> The importer docker_importer indicated a failed response when uploading
>>>>>>>> docker_manifest unit to repository docker_centos7.\n",
>>>>>>>>
>>>>>>>> "error": {
>>>>>>>> "code": "PLP0047",
>>>>>>>> "data": {
>>>>>>>> "unit_type": "docker_manifest",
>>>>>>>> "importer_id": "docker_importer",
>>>>>>>> "repo_id": "docker_centos7",
>>>>>>>> "details": {},
>>>>>>>> "summary": [
>>>>>>>> "Cannot resolve field \"digest\""
>>>>>>>> ]
>>>>>>>> },
>>>>>>>> "description": "The importer docker_importer indicated a failed
>>>>>>>> response when uploading docker_manifest unit to repository
>>>>>>>> docker_centos7.",
>>>>>>>>
>>>>>>>> I tried both v2s2 and v2s1, both don't work. I tried 2 different
>>>>>>>> Dockerfiles and both produce the same error.
>>>>>>>>
>>>>>>>> Is it an issue in Buildah or in Pulp? Or am I doing something wrong?
>>>>>>>>
>>>>>>>> Thanks!
>>>>>>>>
>>>>>>>> --
>>>>>>>> Konstantin Khankin
>>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>> --
>>>>>>> Ханкин Константин
>>>>>>> _______________________________________________
>>>>>>> Pulp-list mailing list
>>>>>>> Pulp-list at redhat.com
>>>>>>> https://www.redhat.com/mailman/listinfo/pulp-list
>>>>>>
>>>>>>
>>>>>
>>>>> --
>>>>> Konstantin Khankin
>>>>> _______________________________________________
>>>>> Pulp-list mailing list
>>>>> Pulp-list at redhat.com
>>>>> https://www.redhat.com/mailman/listinfo/pulp-list
>>>>
>>>>
>>>
>>> --
>>> Konstantin Khankin
>>>
>>
>>
>> --
>> Ханкин Константин
>> _______________________________________________
>> Pulp-list mailing list
>> Pulp-list at redhat.com
>> https://www.redhat.com/mailman/listinfo/pulp-list
>
>
--
Ханкин Константин
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://listman.redhat.com/archives/pulp-list/attachments/20190926/a776f43b/attachment.htm>
More information about the Pulp-list
mailing list