[Pulp-list] Do you remove docker content from the repositories?

[Simon Baatz]

Hi Ina,

On Fri, Jul 19, 2019 at 01:13:20PM +0200, Ina Panova wrote:
>    Hi Simon,
>    these are good observations.
>    It is challenging to find balance between the freedom given to the user
>    and still keeping the repository in a sane state.

I agree, there is no "silver bullet".  Regardless of the approach,
some uses cases will turn out to be more elegant than others.

>    General thinking is:
>    If a docker image manifest is *explicitly* mentioned to be removed by
>    its digest - we perform down recursive removal even if it is referenced
>    by a manifest list.
>    The reason why we also remove  the tags because they are completely
>    useless if they point to something non-existent, meanwhile manifest
>    list is still usable for other remaining image manifests.
>    This exact case we leave on the user's good will, especially if he has
>    provided the digest, meaning 'he knows what he's doing' . There is

I don't know whether we can make this assumption.  Deleting a tag
only deletes that tag.  Thus, the only way to actually remove an
image from a repository is to delete the manifest the tag is pointing
to. In a sense, we force users to delete manifests.

Btw.  we have code in our automation to explicitly address this case,
i.e.  we filter out manifest digests when the manifest digests are
referenced by manifest lists that are still needed.  This code always
felt out of place; Docker is the only artifact type that requires
special treatment when removing artifacts.

>    other side of the coin where the restriction "you cannot delete a
>    manifest that is still referenced by
>    a manifest list or tag" is undesirable. It would force the user to 1)
>    try to remove manifest X and see that you cannot because of tags A,B,C
>    2) removed those tags , 3) only after you can remove manifest X
>    or 1) before head find the tags manifest X references 2) remove tags 3)
>    remove manifest X

I think this depends on the use case.  If "images are identified by
tags" is the common case for users, we can include tags in the
"delete everything on lower level" approach I proposed.  Then, the
user would simply delete tags and Pulp would just take care of the
rest by recursively removing unreferences manifests (lists)/blobs.

The native Docker CLI has a similar approach for images stored
locally.  Additionally, it seems to have the optimization to delete
the tag as well when it is the only tag remaining:

$ docker rmi 59788edf1f3e
Error response from daemon: conflict: unable to delete 59788edf1f3e (must be forced) - image is referenced in multiple repositories
$ docker rmi docker.io/busybox:latest
Untagged: docker.io/busybox:latest
$ docker rmi 59788edf1f3e
Untagged: docker.io/busybox:sometag
Untagged: docker.io/busybox at sha256:2a03a6059f21e150ae84b0973863609494aad70f0a80eaeb64bddd8d92465812
Deleted: sha256:59788edf1f3e78cd0ebe6ce1446e9d10788225db3dedcfd1a59f764bad2b2690
Deleted: sha256:8a788232037eaf17794408ff3df6b922a1aedf9ef8de36afdae3ed0b0381907b

(As said in my previous mail, I don't like the approach to delete
recursively on tag removal. But it may be what most users expect(?))

>    As a possible solution we could add a 'force' flag that will not rely
>    on the users' good will, but will ensure that if a user specifies it he
>    is aware of implications. Otherwise if no 'force' flag is specified we
>    go with behaviour:
>    ---->
>    You can remove an artifact if no
>    artifact on a higher level references it. If it can be removed, all
>    (otherwise unused) artifacts on lower levels are deleted recursively.
>    For example, you cannot delete a manifest that is still referenced by
>    a manifest list or tag.
>    Thoughts?

That sounds like a good idea. Would the recursive removal also be done
on deletion of a tag?