[Pulp-dev] Transition from Mongo to Postgre

[Filip Nguyen]

All that makes sense. Do you have any concerns/benchmarks regarding 
performance after migration? Have you tried migrating your data in some 
POC and measure the difference in some subset of functionality?

Just to give more context. The reason I am asking is that in Candlepin, 
we always have had Postgres as a datastore. Some performance challenges 
that we have been facing:

  * The sheer amount of data on one node makes and the amount of time
    necessary to migrate when changing schema
  * Transaction lock waits - which, at occasion, can stop client
    requests and bring down the service
  * Transaction deadlocks

With every new functionality I sometimes think about a possibility to 
use some NoSQL to make it horizontally scalable. So it would be 
interesting to hear that Mongo didn't deliver the performance in Pulp's 
case.

On 09/13/2016 03:11 PM, Michael Hrivnak wrote:
> We have a thread here about a lot of the 3.0 stack choices, although 
> it seems to skip past the assumption that we're moving to postgres:
>
> https://www.redhat.com/archives/pulp-list/2016-May/msg00042.html
>
> I can't quickly find another summary of why, so I'll describe the 
> highlights here:
>
> - Pulp has highly relational data. The core use case is managing the 
> relationships between content and repositories. Using a relational DB 
> makes that a lot easier.
> - A schemaless DB makes it easy to do writes, but you have to be very 
> careful when doing reads that the your software is prepared for 
> whatever data structure comes out. If you want to enforce a schema, it 
> has to be done in software. It's doable, but requires great care.
> - Transactions!
> - The HA story with mongodb is more complex than most people realize 
> (certainly more complex than we expected). To get real HA with data 
> safety, you have to do a lot of the work in your own software.
>
> MongoDB is great at what it does and a good fit for some use cases, 
> but we learned that it's not the best fit for Pulp.
>
> Michael
>
> On Tue, Sep 13, 2016 at 3:21 AM, Filip Nguyen <fnguyen at redhat.com 
> <mailto:fnguyen at redhat.com>> wrote:
>
>     I heard that Pulp is switching from Mongo to Postgre. Just out of
>     curiosity, I would like to learn more about the reasons why you
>     decided to go this direction. Is there any document/email thread
>     about it?
>
>     _______________________________________________
>     Pulp-dev mailing list
>     Pulp-dev at redhat.com <mailto:Pulp-dev at redhat.com>
>     https://www.redhat.com/mailman/listinfo/pulp-dev
>     <https://www.redhat.com/mailman/listinfo/pulp-dev>
>
>

-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://listman.redhat.com/archives/pulp-dev/attachments/20160914/6c0329be/attachment.htm>