[Pulp-list] pulp 3.3.1 server error

Wed May 13 19:23:51 UTC 2020

Attached are the schema dump from the problematic db fail.sql (3.3.1) and prod db prod.sql (3.2)

Thanks for looking into this.

From: dkliban at redhat.com At: 05/13/20 12:40:42To:  Bin Li (BLOOMBERG/ 120 PARK ) 
Cc:  ipanova at redhat.com,  pulp-list at redhat.com
Subject: Re: [Pulp-list] pulp 3.3.1 server error

Could you create a clone of your production system and try upgrading the cloned environment? I suspect that some problems may have been created from the extra migration being created in between all the upgrade attempts on your test system.

Could you provide the SQL schema (without data) from the database of the system that is experiencing the problem? We want to compare to a clean installed pulp 3.3.1 schema. 

Could you provide the SQL schema (without data) from the database of the 3.2 production system? 

On Wed, May 13, 2020 at 11:52 AM Bin Li (BLOOMBERG/ 120 PARK) <bli111 at bloomberg.net> wrote:

Here is what I have
# django-admin showmigrations
admin
 [X] 0001_initial
 [X] 0002_logentry_remove_auto_add
 [X] 0003_logentry_add_action_flag_choices
auth
 [X] 0001_initial
 [X] 0002_alter_permission_name_max_length
 [X] 0003_alter_user_email_max_length
 [X] 0004_alter_user_username_opts
 [X] 0005_alter_user_last_login_null
 [X] 0006_require_contenttypes_0002
 [X] 0007_alter_validators_add_error_messages
 [X] 0008_alter_user_username_max_length
 [X] 0009_alter_user_last_name_max_length
 [X] 0010_alter_group_name_max_length
 [X] 0011_update_proxy_permissions
contenttypes
 [X] 0001_initial
 [X] 0002_remove_content_type_name
core
 [X] 0001_initial
 [X] 0002_increase_artifact_size_field
 [X] 0003_remove_upload_completed
 [X] 0004_add_duplicated_reserved_resources
 [X] 0005_progressreport_code
 [X] 0006_repository_plugin_managed
 [X] 0007_delete_progress_proxies
 [X] 0008_published_metadata_as_content
 [X] 0009_remove_task_non_fatal_errors
 [X] 0010_pulp_fields
 [X] 0011_relative_path
 [X] 0012_auto_20191104_2000
 [X] 0013_repository_pulp_type
 [X] 0014_remove_repository_plugin_managed
 [X] 0015_auto_20191112_1426
 [X] 0016_charfield_to_textfield
 [X] 0017_remove_task_parent
 [X] 0018_auto_20191127_2350
 [X] 0019_add_signing_service_model
 [X] 0020_change_publishedartifact_constraints
 [X] 0021_add_signing_service_foreign_key
 [X] 0022_rename_last_version
 [X] 0023_change_exporter_models
 [X] 0024_use_local_storage_for_uploads
 [X] 0025_task_parent_task
 [X] 0026_task_group
 [X] 0027_export_backend
 [X] 0028_import_importer_pulpimporter_pulpimporterrepository
file
 [X] 0001_initial
 [X] 0002_file_related_names
 [X] 0003_auto_20191014_1721
 [X] 0004_filefilesystemexporter
 [X] 0005_filerepository
 [X] 0006_delete_filefilesystemexporter
 [X] 0007_filefilesystemexporter
rpm
 [X] 0001_initial
 [X] 0002_updaterecord_reboot_suggested
 [X] 0003_DATA_incorrect_json
 [X] 0004_add_metadata_signing_service_fk
 [X] 0005_optimize_sync
 [X] 0006_opensuse_support
 [X] 0007_checksum_types
 [X] 0008_advisory_pkg_sumtype_as_int
 [X] 0009_revision_null
sessions
 [X] 0001_initial

From: ipanova at redhat.com At: 05/13/20 11:41:13To:  Bin Li (BLOOMBERG/ 120 PARK ) 
Cc:  ttereshc at redhat.com,  pulp-list at redhat.com
Subject: Re: [Pulp-list] pulp 3.3.1 server error

Can you confirm that migration 0009_revision_null was applied?

You can see the migrations with this command 'django-admin showmigrations'

--------
Regards,

Ina Panova
Senior Software Engineer| Pulp| Red Hat Inc.

"Do not go where the path may lead,
 go instead where there is no path and leave a trail."

On Wed, May 13, 2020 at 4:09 PM Bin Li (BLOOMBERG/ 120 PARK) <bli111 at bloomberg.net> wrote:

Hi Ina,

The error was not from migration. I see the error when I try to create a new repo. My pulp_installer is on the latest version. Rerun the installer doesn't solve the issue. What should I do to fix this? We cannot reset database from scratch since we already have a lot data.

From: ipanova at redhat.com At: 05/13/20 09:43:45To:  Bin Li (BLOOMBERG/ 120 PARK ) 
Cc:  ttereshc at redhat.com,  pulp-list at redhat.com
Subject: Re: [Pulp-list] pulp 3.3.1 server error

This issue was fixed in the 3.3.1 release [0][1]

[0] https://pulp-rpm.readthedocs.io/en/latest/changes.html#bugfixes
[1] https://pulp.plan.io/issues/6662

--------
Regards,

Ina Panova
Senior Software Engineer| Pulp| Red Hat Inc.

"Do not go where the path may lead,
 go instead where there is no path and leave a trail."

On Tue, May 12, 2020 at 11:30 PM Bin Li (BLOOMBERG/ 120 PARK) <bli111 at bloomberg.net> wrote:

I removed the 0009_auto.. and rerun the installer. The upgrade was successful.
Below is error log.

May 12 17:25:05 myhost gunicorn[55426]: pulp: django.request:ERROR: Internal Server Error: /pulp/api/v3/repositories/rpm/rpm/
May 12 17:25:05 myhost gunicorn[55426]: Traceback (most recent call last):
May 12 17:25:05 myhost gunicorn[55426]: File "/opt/utils/venv/pulp/3.7.3/lib64/python3.7/site-packages/django/db/backends/utils.py", line 84, in _execute
May 12 17:25:05 myhost gunicorn[55426]: return self.cursor.execute(sql, params)
May 12 17:25:05 myhost gunicorn[55426]: psycopg2.errors.NotNullViolation: null value in column "last_sync_revision_number" violates not-null constraint
May 12 17:25:05 myhost gunicorn[55426]: DETAIL:  Failing row contains (0053634d-d0ca-46de-81d4-b8acf19d592e, f, null, null, 0, null, {}).
May 12 17:25:05 myhost gunicorn[55426]: The above exception was the direct cause of the following exception:
May 12 17:25:05 myhost gunicorn[55426]: Traceback (most recent call last):
May 12 17:25:05 myhost gunicorn[55426]: File "/opt/utils/venv/pulp/3.7.3/lib64/python3.7/site-packages/django/core/handlers/exception.py", line 34, in inner
May 12 17:25:05 myhost gunicorn[55426]: response = get_response(request)
May 12 17:25:05 myhost gunicorn[55426]: File "/opt/utils/venv/pulp/3.7.3/lib64/python3.7/site-packages/django/core/handlers/base.py", line 115, in _get_response
May 12 17:25:05 myhost gunicorn[55426]: response = self.process_exception_by_middleware(e, request)
May 12 17:25:05 myhost gunicorn[55426]: File "/opt/utils/venv/pulp/3.7.3/lib64/python3.7/site-packages/django/core/handlers/base.py", line 113, in _get_response
May 12 17:25:05 myhost gunicorn[55426]: response = wrapped_callback(request, *callback_args, **callback_kwargs)
May 12 17:25:05 myhost gunicorn[55426]: File "/opt/utils/venv/pulp/3.7.3/lib64/python3.7/site-packages/django/views/decorators/csrf.py", line 54, in wrapped_view
May 12 17:25:05 myhost gunicorn[55426]: return view_func(*args, **kwargs)
May 12 17:25:05 myhost gunicorn[55426]: File "/opt/utils/venv/pulp/3.7.3/lib64/python3.7/site-packages/rest_framework/viewsets.py", line 114, in view
May 12 17:25:05 myhost gunicorn[55426]: return self.dispatch(request, *args, **kwargs)
May 12 17:25:05 myhost gunicorn[55426]: File "/opt/utils/venv/pulp/3.7.3/lib64/python3.7/site-packages/rest_framework/views.py", line 505, in dispatch
May 12 17:25:05 myhost gunicorn[55426]: response = self.handle_exception(exc)
May 12 17:25:05 myhost gunicorn[55426]: File "/opt/utils/venv/pulp/3.7.3/lib64/python3.7/site-packages/rest_framework/views.py", line 465, in handle_exception
May 12 17:25:05 myhost gunicorn[55426]: self.raise_uncaught_exception(exc)
May 12 17:25:05 myhost gunicorn[55426]: File "/opt/utils/venv/pulp/3.7.3/lib64/python3.7/site-packages/rest_framework/views.py", line 476, in raise_uncaught_exception
May 12 17:25:05 myhost gunicorn[55426]: raise exc
May 12 17:25:05 myhost gunicorn[55426]: File "/opt/utils/venv/pulp/3.7.3/lib64/python3.7/site-packages/rest_framework/views.py", line 502, in dispatch
May 12 17:25:05 myhost gunicorn[55426]: response = handler(request, *args, **kwargs)
May 12 17:25:05 myhost gunicorn[55426]: File "/opt/utils/venv/pulp/3.7.3/lib64/python3.7/site-packages/rest_framework/mixins.py", line 19, in create
May 12 17:25:05 myhost gunicorn[55426]: self.perform_create(serializer)
May 12 17:25:05 myhost gunicorn[55426]: File "/opt/utils/venv/pulp/3.7.3/lib64/python3.7/site-packages/rest_framework/mixins.py", line 24, in perform_create
May 12 17:25:05 myhost gunicorn[55426]: serializer.save()
May 12 17:25:05 myhost gunicorn[55426]: File "/opt/utils/venv/pulp/3.7.3/lib64/python3.7/site-packages/rest_framework/serializers.py", line 213, in save
May 12 17:25:05 myhost gunicorn[55426]: self.instance = self.create(validated_data)
May 12 17:25:05 myhost gunicorn[55426]: File "/opt/utils/venv/pulp/3.7.3/lib64/python3.7/site-packages/rest_framework/serializers.py", line 932, in create
May 12 17:25:05 myhost gunicorn[55426]: instance = ModelClass._default_manager.create(**validated_data)
May 12 17:25:05 myhost gunicorn[55426]: File "/opt/utils/venv/pulp/3.7.3/lib64/python3.7/site-packages/django/db/models/manager.py", line 82, in manager_method
May 12 17:25:05 myhost gunicorn[55426]: return getattr(self.get_queryset(), name)(*args, **kwargs)
May 12 17:25:05 myhost gunicorn[55426]: File "/opt/utils/venv/pulp/3.7.3/lib64/python3.7/site-packages/django/db/models/query.py", line 422, in create
May 12 17:25:05 myhost gunicorn[55426]: obj.save(force_insert=True, using=self.db)
May 12 17:25:05 myhost gunicorn[55426]: File "/opt/utils/venv/pulp/3.7.3/lib64/python3.7/site-packages/pulpcore/app/models/repository.py", line 62, in save
May 12 17:25:05 myhost gunicorn[55426]: super().save(*args, **kwargs)
May 12 17:25:05 myhost gunicorn[55426]: File "/opt/utils/venv/pulp/3.7.3/lib64/python3.7/site-packages/pulpcore/app/models/base.py", line 107, in save
May 12 17:25:05 myhost gunicorn[55426]: return super().save(*args, **kwargs)
May 12 17:25:05 myhost gunicorn[55426]: File "/opt/utils/venv/pulp/3.7.3/lib64/python3.7/site-packages/django/db/models/base.py", line 741, in save
May 12 17:25:05 myhost gunicorn[55426]: force_update=force_update, update_fields=update_fields)
May 12 17:25:05 myhost gunicorn[55426]: File "/opt/utils/venv/pulp/3.7.3/lib64/python3.7/site-packages/django/db/models/base.py", line 779, in save_base
May 12 17:25:05 myhost gunicorn[55426]: force_update, using, update_fields,
May 12 17:25:05 myhost gunicorn[55426]: File "/opt/utils/venv/pulp/3.7.3/lib64/python3.7/site-packages/django/db/models/base.py", line 870, in _save_table
May 12 17:25:05 myhost gunicorn[55426]: result = self._do_insert(cls._base_manager, using, fields, update_pk, raw)
May 12 17:25:05 myhost gunicorn[55426]: File "/opt/utils/venv/pulp/3.7.3/lib64/python3.7/site-packages/django/db/models/base.py", line 908, in _do_insert
May 12 17:25:05 myhost gunicorn[55426]: using=using, raw=raw)
May 12 17:25:05 myhost gunicorn[55426]: File "/opt/utils/venv/pulp/3.7.3/lib64/python3.7/site-packages/django/db/models/manager.py", line 82, in manager_method
May 12 17:25:05 myhost gunicorn[55426]: return getattr(self.get_queryset(), name)(*args, **kwargs)
May 12 17:25:05 myhost gunicorn[55426]: File "/opt/utils/venv/pulp/3.7.3/lib64/python3.7/site-packages/django/db/models/query.py", line 1186, in _insert
May 12 17:25:05 myhost gunicorn[55426]: return query.get_compiler(using=using).execute_sql(return_id)
May 12 17:25:05 myhost gunicorn[55426]: File "/opt/utils/venv/pulp/3.7.3/lib64/python3.7/site-packages/django/db/models/sql/compiler.py", line 1368, in execute_sql
May 12 17:25:05 myhost gunicorn[55426]: cursor.execute(sql, params)
May 12 17:25:05 myhost gunicorn[55426]: File "/opt/utils/venv/pulp/3.7.3/lib64/python3.7/site-packages/django/db/backends/utils.py", line 67, in execute
May 12 17:25:05 myhost gunicorn[55426]: return self._execute_with_wrappers(sql, params, many=False, executor=self._execute)
May 12 17:25:05 myhost gunicorn[55426]: File "/opt/utils/venv/pulp/3.7.3/lib64/python3.7/site-packages/django/db/backends/utils.py", line 76, in _execute_with_wrappers
May 12 17:25:05 myhost gunicorn[55426]: return executor(sql, params, many, context)
May 12 17:25:05 myhost gunicorn[55426]: File "/opt/utils/venv/pulp/3.7.3/lib64/python3.7/site-packages/django/db/backends/utils.py", line 84, in _execute
May 12 17:25:05 myhost gunicorn[55426]: return self.cursor.execute(sql, params)
May 12 17:25:05 myhost gunicorn[55426]: File "/opt/utils/venv/pulp/3.7.3/lib64/python3.7/site-packages/django/db/utils.py", line 89, in __exit__
May 12 17:25:05 myhost gunicorn[55426]: raise dj_exc_value.with_traceback(traceback) from exc_value
May 12 17:25:05 myhost gunicorn[55426]: File "/opt/utils/venv/pulp/3.7.3/lib64/python3.7/site-packages/django/db/backends/utils.py", line 84, in _execute
May 12 17:25:05 myhost gunicorn[55426]: return self.cursor.execute(sql, params)
May 12 17:25:05 myhost gunicorn[55426]: django.db.utils.IntegrityError: null value in column "last_sync_revision_number" violates not-null constraint
May 12 17:25:05 myhost gunicorn[55426]: DETAIL:  Failing row contains (0053634d-d0ca-46de-81d4-b8acf19d592e, f, null, null, 0, null, {}).
May 12 17:25:05 myhost gunicorn[55426]: 127.0.0.1 - admin [12/May/2020:21:25:05 +0000] "POST /pulp/api/v3/repositories/rpm/rpm/ HTTP/1.1" 500 27 "-" "HTTPie/0.9.4"

From: ttereshc at redhat.com At: 05/12/20 16:19:31To:  Bin Li (BLOOMBERG/ 120 PARK ) 
Cc:  pulp-list at redhat.com
Subject: Re: [Pulp-list] pulp 3.3.1 server error

Was the upgrade successful after you removed the 0009_auto...?  Did you run the installer again?

If no, please do so.
If yes, could you share logs? There will be some traceback showing why 500 error happened.

Thank you,
Tanya

_______________________________________________
Pulp-list mailing list
Pulp-list at redhat.com
https://www.redhat.com/mailman/listinfo/pulp-list

_______________________________________________
Pulp-list mailing list
Pulp-list at redhat.com
https://www.redhat.com/mailman/listinfo/pulp-list

-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://listman.redhat.com/archives/pulp-list/attachments/20200513/07cecf24/attachment.htm>
-------------- next part --------------
A non-text attachment was scrubbed...
Name: fail.sql
Type: application/octet-stream
Size: 137307 bytes
Desc: not available
URL: <http://listman.redhat.com/archives/pulp-list/attachments/20200513/07cecf24/attachment.obj>
-------------- next part --------------
A non-text attachment was scrubbed...
Name: prod.sql
Type: application/octet-stream
Size: 119244 bytes
Desc: not available
URL: <http://listman.redhat.com/archives/pulp-list/attachments/20200513/07cecf24/attachment-0001.obj>