[Pulp-dev] SQlite Issues w/ Pulp3. Switch back to PostgreSQL

Brian Bouterse bbouters at redhat.com
Thu Apr 26 16:50:17 UTC 2018


Thanks to @daviddavis for making PRs resolving both of the issues. Several
core devs ack'd it so I merged this change. If your travis jobs are failing
due to sqlite tests, rebasing will resolve it for you.

Note from @jeremy's link to the isolation docs, it suggests that the WAL
mode allows for multiple writer locks even with multiprocessing. I was
testing using WAL mode yesterday, so the issue was still that the timeout
wasn't working.

If someone is able to get Pulp working perfectly with sqlite please reply
on-list.

On Wed, Apr 25, 2018 at 5:18 PM, Brian Bouterse <bbouters at redhat.com> wrote:

>
>
> On Wed, Apr 25, 2018 at 3:16 PM, Brian Bouterse <bbouters at redhat.com>
> wrote:
>
>> tldr: A bug in sqlite is preventing Pulp from running pulp-smash tests
>> without error making sqlite a non-viable default for Pulp. We should switch
>> back to Postgres as the default for Pulp and its docs, the dev environment,
>> and remove sqlite from the Travis matrix.
>>
>> Q: Why isn't the timeout [0] being applied?
>> A: because it's broken in sqlite's C code (the evidence suggests)
>>
>> Q: What units is the timeout in exactly?
>> A: it's in seconds and our configuration of it was correct. In cPython's
>> code it gets converted to milliseconds which is the right unit for SQLite's
>> interface.
>>
>> Q: If this gets fixed in the future will we switch back?
>> A: Once we enter RC or GA for core we cannot switch the default. So if
>> this gets resolved "real soon" by the sqlite folks then yes, otherwise no.
>>
>> Q: How do you know it's broken?
>> A: I set the timeout to 300 seconds, but the database locked error is
>> still raised after the default of 5 seconds. I use the `PRAGMA
>> busy_timeout;` to confirm that the sqlite knows the 300 second timeout has
>> been set. I've analyzed the django [1] and Python C [2] code also to
>> confirm that our setting is being handled properly all the way down to
>> sqlite's C interface. I've raised the issue with the sqlite community and
>> I'm waiting to hear back.
>>
>> Q: What are the next steps?
>> A: We need issues to switch Pulp's settings and docs to Postgresql.
>> Similarly for switching the devel environment. Also one to remove sqlite
>> from Travis. We need these like real soon.
>>
>
> I think these two issues should be groomed and added to the sprint. There
> is outstanding that can't merge because of this issue causes Travis tests
> to fail which blocks merging.
>
> https://pulp.plan.io/issues/3606
> https://pulp.plan.io/issues/3505
>
>
>>
>> Q: What about the community's concerns?
>> A: They can help contribute to the sqlite bug that is prevent Pulp from
>> being meaningfully used on top of sqlite. They can still configure sqlite
>> and use it just like any django supported db, we just don't test on top of
>> it on Travis anymore. It is erroring almost all of our Travis jobs.
>>
>> In terms of the how we document PostgreSQL, I want to advocate avoiding
>> documenting command by command installs of Postgres. I used to feel
>> differently about this, but I think if users start on our docs and they
>> don't work, we're probably not going to help them get their Postgresql
>> going on some random distro. Soon the Ansible installer will solve this for
>> our users anyway so I don't think it's a big problem.
>>
>> Ideas, concerns, help writing issues, and questions are all welcome!
>>
>> [0]: https://docs.djangoproject.com/en/1.11/ref/databases/#databa
>> se-is-locked-errors
>> [1]: https://github.com/django/django/blob/1.11.12/django/db/back
>> ends/sqlite3/base.py#L177
>> [2]: https://github.com/python/cpython/blob/master/Modules/_sqlit
>> e/connection.c#L181
>>
>> -Brian
>>
>>
>>
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://listman.redhat.com/archives/pulp-dev/attachments/20180426/27d5414e/attachment.htm>


More information about the Pulp-dev mailing list