[Pulp-list] working directories proposal

[Michael Hrivnak]

----- Original Message -----
> From: "Dennis Kliban" <dkliban at redhat.com>
> To: "pulp-list" <pulp-list at redhat.com>
> Sent: Monday, December 15, 2014 2:37:35 PM
> Subject: [Pulp-list] working directories proposal
> 
> What we have now:
> 
>   - Storage directory defined here [0]
> 
>   - The above value is used to create a path for working directories here [1]
> 
> Proposed changes:
> 
>   - Add a new config value in the 'server' section called
>   'working_directory'.  It's default value would be /var/lib/storage

We should have "pulp" in the path. /var/cache/pulp would be a good choice and works well with the FHS.

> 
>   - Change common utils [1] to use ['server']['working_dirctory'] config as
>   base path.
> 
>   - Create 'working_directories' collection in database.
> 
>   - Add a check to repository_working_dir [2] and the rest of *_working_dir
>   methods to determine if this method is called from a task.  If it is, add
>   the task id, worker id, and path to 'working_directories' collection.
> 
>   - Create a period task that will check for any tasks in final state with
>   existing working directories and delete them.  This will need to take into
>   account that the directories exist on a specific worker.

This may be complicated. The directory exists on a specific machine, which may have several workers. Dispatching the cleanup task to a specific worker or specific machine may be difficult.

Here's another option that wouldn't require a DB collection. Assume that /var/cache/pulp/ will actually be the value of the "working_directory" setting.

When a worker starts, it creates the directory /var/cache/pulp/<worker_id>/. If it already exists, any content gets blown away. On exit, the whole directory named <worker_id> gets deleted.

If a task requests temporary storage, the worker creates /var/cache/pulp/<worker_id>/<task_id>/ and makes a new temp directory within that. Multiple directories can be requested per task. When the task completes, the worker blows away the entire directory named <task_id>.

I think that covers all of our bases. Thoughts?