[Pulp-dev] SQlite Issues w/ Pulp3. Switch back to PostgreSQL

Brian Bouterse bbouters at redhat.com
Thu Apr 26 16:50:17 UTC 2018

Thanks to @daviddavis for making PRs resolving both of the issues. Several
core devs ack'd it so I merged this change. If your travis jobs are failing
due to sqlite tests, rebasing will resolve it for you.

Note from @jeremy's link to the isolation docs, it suggests that the WAL
mode allows for multiple writer locks even with multiprocessing. I was
testing using WAL mode yesterday, so the issue was still that the timeout
wasn't working.

If someone is able to get Pulp working perfectly with sqlite please reply

On Wed, Apr 25, 2018 at 5:18 PM, Brian Bouterse <bbouters at redhat.com> wrote:

> On Wed, Apr 25, 2018 at 3:16 PM, Brian Bouterse <bbouters at redhat.com>
> wrote:
>> tldr: A bug in sqlite is preventing Pulp from running pulp-smash tests
>> without error making sqlite a non-viable default for Pulp. We should switch
>> back to Postgres as the default for Pulp and its docs, the dev environment,
>> and remove sqlite from the Travis matrix.
>> Q: Why isn't the timeout [0] being applied?
>> A: because it's broken in sqlite's C code (the evidence suggests)
>> Q: What units is the timeout in exactly?
>> A: it's in seconds and our configuration of it was correct. In cPython's
>> code it gets converted to milliseconds which is the right unit for SQLite's
>> interface.
>> Q: If this gets fixed in the future will we switch back?
>> A: Once we enter RC or GA for core we cannot switch the default. So if
>> this gets resolved "real soon" by the sqlite folks then yes, otherwise no.
>> Q: How do you know it's broken?
>> A: I set the timeout to 300 seconds, but the database locked error is
>> still raised after the default of 5 seconds. I use the `PRAGMA
>> busy_timeout;` to confirm that the sqlite knows the 300 second timeout has
>> been set. I've analyzed the django [1] and Python C [2] code also to
>> confirm that our setting is being handled properly all the way down to
>> sqlite's C interface. I've raised the issue with the sqlite community and
>> I'm waiting to hear back.
>> Q: What are the next steps?
>> A: We need issues to switch Pulp's settings and docs to Postgresql.
>> Similarly for switching the devel environment. Also one to remove sqlite
>> from Travis. We need these like real soon.
> I think these two issues should be groomed and added to the sprint. There
> is outstanding that can't merge because of this issue causes Travis tests
> to fail which blocks merging.
> https://pulp.plan.io/issues/3606
> https://pulp.plan.io/issues/3505
>> Q: What about the community's concerns?
>> A: They can help contribute to the sqlite bug that is prevent Pulp from
>> being meaningfully used on top of sqlite. They can still configure sqlite
>> and use it just like any django supported db, we just don't test on top of
>> it on Travis anymore. It is erroring almost all of our Travis jobs.
>> In terms of the how we document PostgreSQL, I want to advocate avoiding
>> documenting command by command installs of Postgres. I used to feel
>> differently about this, but I think if users start on our docs and they
>> don't work, we're probably not going to help them get their Postgresql
>> going on some random distro. Soon the Ansible installer will solve this for
>> our users anyway so I don't think it's a big problem.
>> Ideas, concerns, help writing issues, and questions are all welcome!
>> [0]: https://docs.djangoproject.com/en/1.11/ref/databases/#databa
>> se-is-locked-errors
>> [1]: https://github.com/django/django/blob/1.11.12/django/db/back
>> ends/sqlite3/base.py#L177
>> [2]: https://github.com/python/cpython/blob/master/Modules/_sqlit
>> e/connection.c#L181
>> -Brian
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://listman.redhat.com/archives/pulp-dev/attachments/20180426/27d5414e/attachment.htm>

More information about the Pulp-dev mailing list