[Pulp-list] recover from filesystem full

[Michael Hrivnak]

Alex,

I'm glad you were able to get things back up and running. So would you summarize your path to success as 1) free up some space, 2) restart services, and then everything was fine? Did you have to do anything with mongodb?

FWIW, my favorite way to quickly free up space and inodes is "yum clean all".

There was discussion of this recently, that we need to investigate all the ways that pulp could fail when the filesystem fills up. Presumably most of that would happen during a sync process, and ideally pulp would be able to gracefully stop what it's doing and fail the task.

If someone, perhaps yourself, is willing to spend a little time experimenting with causing a disk-full situation and collecting info about how pulp behaves, and where it got stuck, that would be very helpful! I and others would be happy to brainstorm on how to setup a VM for this, what data to collect, etc.

Michael

----- Original Message -----
From: "Alex Harvey" <alexharv074 at gmail.com>
To: pulp-list at redhat.com
Sent: Thursday, March 19, 2015 9:57:52 PM
Subject: Re: [Pulp-list] recover from filesystem full

OK - I got past this by deleting the RPMs and then restarting all the services. Still, it would be good to know how to recover from a full filesystem in general, without deleting RPMs and then deleting a repository. 

On 20 March 2015 at 12:03, Alex Harvey < alexharv074 at gmail.com > wrote: 



Hi all, 

I've filled up the filesystem in /var/lib/pulp. I can't find any instructions on how to recover from this situation. I tried just deleting all the RPMs that I had downloaded but there's evidently some corruption relating to the filesystem filling. 

I just see this 



# pulp-admin -u admin -p admin rpm repo delete --repo-id centos-6.5-x86_64 

This command may be exited via ctrl+c without affecting the request. 

[-] 


Running... 

[|] 

Waiting to begin... 

Hangs forever. Logs say 



# tail -0f /var/log/messages 

Mar 20 10:25:16 localhost pulp: celery.worker.strategy:INFO: Received task: pulp.server.async.tasks._reserve_resource[cd24998e-c57a-43be-bcea-a4b4f2f6410c] 

Mar 20 10:25:16 localhost pulp: celery.worker.job:INFO: Task pulp.server.async.tasks._reserve_resource[cd24998e-c57a-43be-bcea-a4b4f2f6410c] succeeded in 0.047468271805s: u'reserved_resource_worker-0 at ainslie.ae.sda.corp.telstra.com.dq' 

Mar 20 10:25:16 localhost pulp: celery.worker.strategy:INFO: Received task: pulp.server.tasks.repository.delete[eb2c073f-69f9-4b77-b3fc-cff12e936a7d] 
Best regards 
Alex 


_______________________________________________
Pulp-list mailing list
Pulp-list at redhat.com
https://www.redhat.com/mailman/listinfo/pulp-list