Yum 2.2 w/ HTTP proxy fails w/ error "Cannot find a valid baseurl for repo:"

[Mike Fleetwood]

On Wed, 30 Mar 2005, Paul Howarth wrote:

> On Wed, 2005-03-30 at 12:46 -0800, Kam Leo wrote:
>> On Wed, 30 Mar 2005 08:40:09 +0100, Paul Howarth <paul at city-fan.org> wrote:
>>> On Tue, 2005-03-29 at 22:35 +0100, Mike Fleetwood wrote:
>>>> Recently "yum check-update" and "yum update" has started producing the
>>>> error "Cannot find a valid baseurl for repo: ...".  Googling suggests
>>>> that other occurrences have been caused by network connectivity errors
>>>> or miss configured yum repository locations.  I am convinced that in my
>>>> case it is yum 2.2's inability to successfully download files through a
>>>> HTTP proxy that is the problem.  To work around the problem I
>>>> reinstalled yum 2.1 and upgrade using "yum --exclude=yum update".
>>>
>>> yum 2.2 provides a way to specify the proxy globally or separately for
>>> each repo if required (see "man yum.conf"). Did you try that?
>>
>> The OP commented out the baseurl line in
>> /et/cyum.repos.d/fedora-updates.repo leaving only mirrorlist as the
>> repositoryid.  Per 'man yum.conf':
>
> This is perfectly normal. The baseurl is in fact commented out by
> default. See:
> http://www.mirrorservice.org/sites/download.fedora.redhat.com/pub/fedora/linux/core/3/i386/os/Fedora/RPMS/fedora-release-3-8.i386.rpm//etc/yum.repos.d/fedora-updates.repo?extract=true
>
>> Either the mirrorlist list is defective or there is a bug in yum-2.2.
>
> Clearly the proxy configuration using environment variables, which used
> to work, is no longer doing so. I would however be interested to hear if
> the yum.conf way of specifying proxies now works in yum 2.2.

Executive summary:
I was partially wrong.  Yum only doesn't work properly with HTTP
proxies requiring authorisation.  Yum 2.1 copes, 2.2 doesn't.  Not
using mirrorlist _might_ work around it.


Details:
I have been doing lots of packet tracing, discovery bad assumptions
I have made, and found something closer to my actual problem with yum.

My initial problem was with yum 2.2 on FC3 at work where I have to use
a Microsoft ISA web proxy with password authentication.  Yum 2.1 often
reported errors left, right and centre but would eventually succeed in
downloading the required RPMs.  When I upgraded to yum 2.2 it never
succeeded and still doesn't.  More on this later ...

Flash back to home were I did some brief testing on FC3 which is behind
Stone (http://www.gcd.org/sengoku/stone/) web proxy.  It must have
failed for network problems a couple of times and I assumed that yum
2.2 was the problem.  Sorry for that.  Yum 2.1 and 2.2 successfully
work though this web proxy which doesn't require authentication.

Packet tracing of yum 2.1 and 2.2 using HTTP_PROXY environment variable
to define the Stone web proxy, which doesn't require authentication,
shows that yum works OK.  Yum 2.1 actually downloads the mirror list
for all repos even if they aren't enabled.  It does only download
repomd.xml for enable repos though.  Yum 2.2 only gets the mirror list
for repos that are actually enabled.  An improvement.

Now on to packet tracing of yum 2.1 and 2.2 at work through my
companies MS ISA web proxy which requires authentication ...
I have found that yum doesn't send the HTTP headers correctly when yum
is using an authenticating web proxy.  This is when defining the web
proxy using something like:
   export HTTP_PROXY=http://username:password@webproxy.example.com:8080/
MS ISA web proxy offers to keep the TCP connection open so it can be
used to send more HTTP request over, with the "Keep-Alive" header
which yum tries to use.  A typical TCP conversation goes like this:
   yum>    GET ...
   yum>    Host: ...
   yum>    Accept-Encoding: identity
   yum>    Proxy-authorization: Basic xxxxxxxxxxxxxxxxx
   yum>                      <--- Blank line which shouldn't be here
   yum>    User-agent: urlgrabber/2.9.2
   yum>
   proxy<  HTTP/1.1 200 OK
   proxy<  ...
   proxy<  Content-Length: ...
   proxy<  ...
   proxy<  Keep-Alive: ...
   proxy<
   proxy<  [Requested URL contents ...]
   proxy<  ...
   proxy<  HTTP/1.1 400 Bad Request ( The data is invalid. )
   proxy<  ...
   proxy<  Connection: close
   proxy<  Proxy-Connection: close
   proxy<  ...
   yum>    GET ...
   yum>    Host: ...
   yum>    Accept-Encoding: identity
   yum>    Proxy-authorization: Basic xxxxxxxxxxxxxxxxx
   yum>                      <--- Blank line which shouldn't be here
   yum>    User-agent: urlgrabber/2.9.2
   yum>
And the connection closes or times out.

So Yum (or urlgrabber) is incorrectly inserting a new line after the
Proxy-authorization header before adding the User-agent header.  MS
ISA web proxy thinks that it's 2 separate requests, the first works and
the second results in an error message.  Yum doesn't realise as it only
reads the specified number of bytes for its first request.  It then
reuses the connection, sending another request.  It reads the
previously buffer / unread data and see the error from the proxy.  The
difference between yum 2.1 and 2.2 is that the particular files which
they try to read from closed sockets differ, probably because yum 2.1
reads mirror lists for repos its not using and yum 2.2 doesn't.  Yum
2.1 only ever sees the failures on files it can and does re-try, so it
eventually succeeds.  Where as as yum 2.2 sees the failures on getting
the mirror lists which it doesn't re-try so always fails.

This explanation suggests not using mirrorlist parameter and switching
back to baseurl instead as a possible work around for yum 2.2.  I will
try this in a later.

With all this I have only done one or two tests with yum 2.2 specifying
proxy parameters in the configuration file.  Results seem to indicate
that it just ignores these but don't count on this being the truth.

Many thanks,
Mike
-- 
  __  __ _ _     ___    ____ _    ___  ___  _           ___  ___    _
|  \/  (_| | _ / _ \  | ___| |  / _ \/ _ \| |_ _  _  _/   \/   \ _| |
| |\/| | | |/ |  ___| | _| | |_|  __|  ___| __| \/ \/|  O |  O  / _ |
|_|  |_|_|_|\_\\___|  |_|  |____\___|\___||____\_/^\_/\___/\___/\___|