InstantMirror needs a rethink

[Warren Togami]

Today InstantMirror is pretty useful for home and small office mirrors,
but its limitations make it unsustainable without manual intervention of
the sysadmin.

I've been beginning to think that perhaps InstantMirror is heading down
the wrong path and we seriously need to rethink it.  There are simply
too many limitations of the current "stateless" operation of
InstantMirror where it runs only on-demand as mod_python script:

- Synchronization/locking of multiple connections downloading the same
file is awkward and broken.
- There is no good way to clean up aborted tmp files.
- There is no good way to know what are old files that need pruning.
- There is no good way of keeping track of the "Big Picture" of its own
cache, "least recently used" knowing what files were unpopular locally
and should be pruned.

https://fedorahosted.org/InstantMirror/wiki/InstantMirrorDaemon
We need a daemon to handle all this.  Perhaps the daemon could allow
socket connections from a mod_python script for accesses.  Or perhaps it
might be better for the daemon itself to handle serving connections.

Stepping back, what we really need is:
A reverse proxy caching server with all the logic of squid or varnish,
except it stores its cache with file and directory names intact.

How do we get there?
1) Write a new daemon from scratch?
2) Write a new backend storage engine for squid or varnish?  (Store
files in target directory structure, store metadata elsewhere.)
3) ???

Any thoughts?

Warren Togami
wtogami at redhat.com