[Pulp-list] pulp2 migration: AccessPolicy matching query does not exist

Tatiana Tereshchenko ttereshc at redhat.com
Thu Oct 15 12:09:03 UTC 2020 

As for the teardown method, it was written for use with our tests and is
not exposed as a command in any way.
If it's helpful, here is the list of tables which are NOT SAFE to remove
data from
https://github.com/pulp/pulp-2to3-migration/blob/master/pulp_2to3_migration/tests/functional/constants.py#L1-L14
.

I hope we can provide the reset option soon, and you'll no longer need any
workarounds.

Tanya

On Thu, Oct 15, 2020 at 2:02 PM Tatiana Tereshchenko <ttereshc at redhat.com>
wrote:

> Good to know that it's not for the upgrade.
>
> We've planned to have a way to restart migration from scratch but never
> got to it and no one asked before.
> Here is a story I filed, feel free to provide any feedback there
> https://pulp.plan.io/issues/7714.
>
> It would be helpful for us to understand your use case.
> Could you share why you wanted to start completely from scratch and not
> just re-run the existing migration plan or run a new one?
>
> Thanks,
> Tanya
>
> On Wed, Oct 14, 2020 at 7:07 PM Winberg Adam <Adam.Winberg at smhi.se> wrote:
>
>> > probably because we never ask to wipe out the database for upgrades.
>>
>>
>> My reason for doing a 'flush' was to rerun my pulp2 migration from
>> scratch, so not really because of the upgrade. But with the background
>> provided by you, issue https://pulp.plan.io/issues/6963 now makes more
>> sense to me. I guess I should preferably use the 'teardown' util mentioned
>> there instead? It is however unclear to me how to use that.
>>
>>
>> The 'reset_db' won't work for me since we have a centralized postgres
>> infrastructure where I don't have permissions to drop/create db's (I had to
>> get the help of a DBMS to drop my pulp-db).
>>
>>
>> @Brian Bouterse  - thanks for creating the issue!
>>
>>
>> //Adam
>>
>>
>>
>>
>>
>> ------------------------------
>> *From:* Brian Bouterse <bmbouter at redhat.com>
>> *Sent:* 14 October 2020 18:17
>> *To:* Tatiana Tereshchenko
>> *Cc:* Winberg Adam; pulp-list at redhat.com
>> *Subject:* Re: [Pulp-list] pulp2 migration: AccessPolicy matching query
>> does not exist
>>
>> I've filed this issue tracking the improvement which would allow users to
>> run `flush` and not experience this problem.
>>
>> https://pulp.plan.io/issues/7710
>>
>> On Wed, Oct 14, 2020 at 12:14 PM Tatiana Tereshchenko <
>> ttereshc at redhat.com> wrote:
>>
>>> Adam,
>>> I agree we lack documentation on resetting the environment and the
>>> database, probably because we never ask to wipe out the database for
>>> upgrades.
>>> The instructions are usually provided with release and ask you basically
>>> to run the latest pulp_installer.
>>>
>>> There is some data which is provided with migrations and which has to be
>>> present in the database, that's why dropping the database and applying
>>> migrations work and `flush` does not.
>>> `flush` just removes data and keeps all the tables in place, so
>>> migrations are not re-applied.
>>> So for now, please do not use `flush` if you want to start using pulp 3
>>> db from scratch. We potentially can provide a separate command or find some
>>> other way to fill in the essential data back.
>>> Please file an issue in our tracker if such feature/command is helpful
>>> for you. https://pulp.plan.io/projects/pulp/issues/new
>>>
>>> As a side note, if you are interested, reset_db command drops database.
>>> It's provided as a part of django-extensions package
>>> https://django-extensions.readthedocs.io/en/latest/command_extensions.html
>>>
>>> I'm glad that the migration  runs for you now,
>>> Tanya
>>>
>>>
>>> On Wed, Oct 14, 2020 at 3:52 PM Winberg Adam <Adam.Winberg at smhi.se>
>>> wrote:
>>>
>>>> Thanks for the reply!
>>>>
>>>>
>>>> I use 'flush' to clear the db, I don't have a 'reset_db' command.
>>>> Otherwise that's pretty much my process.
>>>>
>>>>
>>>> The migrate command returns 'No migrations to apply'.
>>>>
>>>> Running 'pulpcore-manager show-migrations' shows that all migrations,
>>>> including 'guardian.0001/guardian.0002' has checkmarks ([X]).
>>>>
>>>> guardian
>>>>  [X] 0001_initial
>>>>  [X] 0002_generic_permissions_index
>>>>
>>>> The creation of the migration plan works so I assume my admin user is
>>>> ok.
>>>>
>>>>
>>>> I am using an rpm based installation on RHEL8, with
>>>>
>>>> python3-pulp-2to3-migration-0.4.0-1.el8.noarch
>>>> python3-pulp-rpm-3.7.0-1.el8.noarch
>>>> python3-pulpcore-3.7.1-3.el8.noarch
>>>>
>>>>
>>>> I don't know what went wrong, but I surrendered and dropped my DB and
>>>> redid the migrations from scratch - and now it works.
>>>> Is there a documented instruction on upgrading existing installations?
>>>>
>>>> //Adam
>>>>
>>>>
>>>>
>>>> ------------------------------
>>>> *From:* Tatiana Tereshchenko <ttereshc at redhat.com>
>>>> *Sent:* 14 October 2020 14:15
>>>> *To:* Winberg Adam
>>>> *Cc:* pulp-list at redhat.com
>>>> *Subject:* Re: [Pulp-list] pulp2 migration: AccessPolicy matching
>>>> query does not exist
>>>>
>>>>
>>>>
>>>> On Wed, Oct 14, 2020 at 2:12 PM Tatiana Tereshchenko <
>>>> ttereshc at redhat.com> wrote:
>>>>
>>>>> Hi Adam,
>>>>>
>>>>> My understanding is that you did the following:
>>>>>  * stop pulp services
>>>>>  * pulpcore-manager (or django-admin) reset_db
>>>>>  * pulpcore-manager migrate
>>>>>  * pulpcore-manager reset-admin-password --password password
>>>>>  * start services
>>>>>  * http POST :/pulp/api/v3/migration-plans/ < your_migraiton_plan.json
>>>>>  * http POST
>>>>> :/pulp/api/v3/migration-plans/48d03a72-96a1-4d36-9f8b-9a57e97846ef/run/
>>>>>
>>>>
>>>> Sent too early :)
>>>> I can't reproduce it so far, so any hints about what can be special
>>>> about your environment or installation would be appreciated.
>>>> Make sure that you have at least one user which has admin
>>>> privileges and that the guardian migrations ran indeed.
>>>>   Applying guardian.0001_initial... OK
>>>>   Applying guardian.0002_generic_permissions_index... OK
>>>>
>>>> Tanya
>>>>
>>>>
>>>>
>>>>
>>>>
>>>>>
>>>>> On Wed, Oct 14, 2020 at 8:02 AM Winberg Adam <Adam.Winberg at smhi.se>
>>>>> wrote:
>>>>>
>>>>>> Hello,
>>>>>>
>>>>>>
>>>>>> so I updated my pulp3 installation from 3.4 to 3.7 and tried to rerun
>>>>>> my pulp2 migration - but it errors out with "AccessPolicy matching
>>>>>> query does not exist". Anyone know why?
>>>>>>
>>>>>>
>>>>>> I flushed my db, reran the 'migrate' job, created a pulp2migration
>>>>>> plan (which worked fine) and then tried to run it. Here's the complete
>>>>>> error:
>>>>>>
>>>>>>
>>>>>> Oct 14 05:43:26  gunicorn[2150852]: pulp: django.request:ERROR:
>>>>>> Internal Server Error:
>>>>>> /pulp/api/v3/migration-plans/48d03a72-96a1-4d36-9f8b-9a57e97846ef/run/
>>>>>> Oct 14 05:43:26  gunicorn[2150852]: Traceback (most recent call last):
>>>>>> Oct 14 05:43:26  gunicorn[2150852]:   File
>>>>>> "/usr/lib/python3.6/site-packages/django/core/handlers/exception.py", line
>>>>>> 34, in inner
>>>>>> Oct 14 05:43:26  gunicorn[2150852]:     response =
>>>>>> get_response(request)
>>>>>> Oct 14 05:43:26  gunicorn[2150852]:   File
>>>>>> "/usr/lib/python3.6/site-packages/django/core/handlers/base.py", line 115,
>>>>>> in _get_response
>>>>>> Oct 14 05:43:26  gunicorn[2150852]:     response =
>>>>>> self.process_exception_by_middleware(e, request)
>>>>>> Oct 14 05:43:26  gunicorn[2150852]:   File
>>>>>> "/usr/lib/python3.6/site-packages/django/core/handlers/base.py", line 113,
>>>>>> in _get_response
>>>>>> Oct 14 05:43:26  gunicorn[2150852]:     response =
>>>>>> wrapped_callback(request, *callback_args, **callback_kwargs)
>>>>>> Oct 14 05:43:26  gunicorn[2150852]:   File
>>>>>> "/usr/lib/python3.6/site-packages/django/views/decorators/csrf.py", line
>>>>>> 54, in wrapped_view
>>>>>> Oct 14 05:43:26  gunicorn[2150852]:     return view_func(*args,
>>>>>> **kwargs)
>>>>>> Oct 14 05:43:26  gunicorn[2150852]:   File
>>>>>> "/usr/lib/python3.6/site-packages/rest_framework/viewsets.py", line 114, in
>>>>>> view
>>>>>> Oct 14 05:43:26  gunicorn[2150852]:     return self.dispatch(request,
>>>>>> *args, **kwargs)
>>>>>> Oct 14 05:43:26  gunicorn[2150852]:   File
>>>>>> "/usr/lib/python3.6/site-packages/rest_framework/views.py", line 505, in
>>>>>> dispatch
>>>>>> Oct 14 05:43:26  gunicorn[2150852]:     response =
>>>>>> self.handle_exception(exc)
>>>>>> Oct 14 05:43:26  gunicorn[2150852]:   File
>>>>>> "/usr/lib/python3.6/site-packages/rest_framework/views.py", line 465, in
>>>>>> handle_exception
>>>>>> Oct 14 05:43:26  gunicorn[2150852]:
>>>>>>  self.raise_uncaught_exception(exc)
>>>>>> Oct 14 05:43:26  gunicorn[2150852]:   File
>>>>>> "/usr/lib/python3.6/site-packages/rest_framework/views.py", line 476, in
>>>>>> raise_uncaught_exception
>>>>>> Oct 14 05:43:26  gunicorn[2150852]:     raise exc
>>>>>> Oct 14 05:43:26  gunicorn[2150852]:   File
>>>>>> "/usr/lib/python3.6/site-packages/rest_framework/views.py", line 502, in
>>>>>> dispatch
>>>>>> Oct 14 05:43:26  gunicorn[2150852]:     response = handler(request,
>>>>>> *args, **kwargs)
>>>>>> Oct 14 05:43:26  gunicorn[2150852]:   File
>>>>>> "/usr/lib/python3.6/site-packages/pulp_2to3_migration/app/viewsets.py",
>>>>>> line 85, in run
>>>>>> Oct 14 05:43:26  gunicorn[2150852]:     'dry_run': dry_run
>>>>>> Oct 14 05:43:26  gunicorn[2150852]:   File
>>>>>> "/usr/lib/python3.6/site-packages/pulpcore/tasking/tasks.py", line 236, in
>>>>>> enqueue_with_reservation
>>>>>> Oct 14 05:43:26  gunicorn[2150852]:     **parent_kwarg,
>>>>>> Oct 14 05:43:26  gunicorn[2150852]:   File
>>>>>> "/usr/lib/python3.6/site-packages/django/db/models/manager.py", line 82, in
>>>>>> manager_method
>>>>>> Oct 14 05:43:26  gunicorn[2150852]:     return
>>>>>> getattr(self.get_queryset(), name)(*args, **kwargs)
>>>>>> Oct 14 05:43:26  gunicorn[2150852]:   File
>>>>>> "/usr/lib/python3.6/site-packages/django/db/models/query.py", line 422, in
>>>>>> create
>>>>>> Oct 14 05:43:26  gunicorn[2150852]:     obj.save(force_insert=True,
>>>>>> using=self.db)
>>>>>> Oct 14 05:43:26  gunicorn[2150852]:   File
>>>>>> "/usr/lib/python3.6/site-packages/django_lifecycle/mixins.py", line 132, in
>>>>>> save
>>>>>> Oct 14 05:43:26  gunicorn[2150852]:
>>>>>>  self._run_hooked_methods(AFTER_CREATE)
>>>>>> Oct 14 05:43:26  gunicorn[2150852]:   File
>>>>>> "/usr/lib/python3.6/site-packages/django_lifecycle/mixins.py", line 207, in
>>>>>> _run_hooked_methods
>>>>>> Oct 14 05:43:26  gunicorn[2150852]:     method()
>>>>>> Oct 14 05:43:26  gunicorn[2150852]:   File
>>>>>> "/usr/lib/python3.6/site-packages/django_lifecycle/decorators.py", line 69,
>>>>>> in func
>>>>>> Oct 14 05:43:26  gunicorn[2150852]:     hooked_method(*args, **kwargs)
>>>>>> Oct 14 05:43:26  gunicorn[2150852]:   File
>>>>>> "/usr/lib/python3.6/site-packages/pulpcore/app/models/access_policy.py",
>>>>>> line 60, in add_perms
>>>>>> Oct 14 05:43:26  gunicorn[2150852]:     access_policy =
>>>>>> AccessPolicy.objects.get(viewset_name=self.ACCESS_POLICY_VIEWSET_NAME)
>>>>>> Oct 14 05:43:26  gunicorn[2150852]:   File
>>>>>> "/usr/lib/python3.6/site-packages/django/db/models/manager.py", line 82, in
>>>>>> manager_method
>>>>>> Oct 14 05:43:26  gunicorn[2150852]:     return
>>>>>> getattr(self.get_queryset(), name)(*args, **kwargs)
>>>>>> Oct 14 05:43:26  gunicorn[2150852]:   File
>>>>>> "/usr/lib/python3.6/site-packages/django/db/models/query.py", line 408, in
>>>>>> get
>>>>>> Oct 14 05:43:26  gunicorn[2150852]:     self.model._meta.object_name
>>>>>> Oct 14 05:43:26  gunicorn[2150852]:
>>>>>> pulpcore.app.models.access_policy.AccessPolicy.DoesNotExist: AccessPolicy
>>>>>> matching query does not exist.
>>>>>>
>>>>>>
>>>>>> Regards
>>>>>>
>>>>>> //Adam
>>>>>> _______________________________________________
>>>>>> Pulp-list mailing list
>>>>>> Pulp-list at redhat.com
>>>>>> https://www.redhat.com/mailman/listinfo/pulp-list
>>>>>
>>>>> _______________________________________________
>>> Pulp-list mailing list
>>> Pulp-list at redhat.com
>>> https://www.redhat.com/mailman/listinfo/pulp-list
>>
>>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://listman.redhat.com/archives/pulp-list/attachments/20201015/48228d08/attachment.htm>

More information about the Pulp-list mailing list