[Libguestfs] [PATCH nbdkit] curl: Add effective-url flag

[Eric Blake]

On Wed, Oct 13, 2021 at 11:06:03AM +0100, Richard W.M. Jones wrote:
> +=item B<effective-url=true>
> +
> +(nbdkit E<ge> 1.30)
> +
> +Replace the URL supplied on the command line with the effective URL
> +(from L<CURLINFO_EFFECTIVE_URL(3)>, the final URL fetched after server
> +redirects).  This can be used with mirror services that redirect to a
> +geographical region — for example file download sites — to ensure the
> +URL will always be the same.
> +
> +Note use of this feature in long-lived nbdkit instances can cause
> +subtle problems:
> +
> +=over 4
> +
> +=item *
> +
> +The effective URL persists across connections for the lifetime of the
> +nbdkit instance.  If nbdkit is used for a long time then it is
> +possible for the redirected URL to become stale.

I know you're already hesitant about adding this, because it adds
complexity for not much gain, and the following suggestion just adds
even more complexity.  But would it be worth setting up some sort of
timeout mechanism, where the user specifies a timeout for how long a
redirect remains resolved?  For example, a user could set up a
redirect lifetime of 1h; if we detect a redirect, then for the next
hour we stick to just the resolved URL, but after 1h elapses, we go
back to probing the original URL (which may now redirect differently).
That may reduce some of the negative effects in a long-running
process.

> +
> +=item *
> +
> +It will defeat some mirror load-balancing techniques.
> +
> +=item *
> +
> +If the mirror service sometimes redirects to a broken URL and it
> +happens that the URL you fetch first is broken then nbdkit will no
> +longer recover on subsequent connections (instead you will need to
> +restart nbdkit).

And for this, we would just treat any error as an instant expiry of
the redirect window timeout (so if we detected a redirect, and
encounter any connection error at all, the next action tries to
re-resolve the original URL and does not pass on failure to the caller
unless the redirect still ended up at the place where we just detected
a failure).

Of course, all that proposed complexity is just thoughts for what we
might change; I don't have a patch to actually implement it.

-- 
Eric Blake, Principal Software Engineer
Red Hat, Inc.           +1-919-301-3266
Virtualization:  qemu.org | libvirt.org