RHEL 4 Update Procedure

[George Magklaras]

I think that your account manager (hardware or RedHat) should be your 
first point of contact. If you don't have one, and you are running 
mission critical servers (I have no idea what size of organization you 
work for) then contact the vendor directly.

Where I work, we do the test and provide the assurance . Our mission 
critical servers are file-web (SAMBA-APACHE servers) and RDBMS servers 
(Postgres, MySQL). So, we have set to the side an identical (excluding 
amount of RAM and number of disks) hardware config, which is used 
especially for this particular purpose. Having a scaled down replica of 
the mission critical boxes for the purposes of trying to see if an 
update breaks something is worth the extra money when compared to the 
downtime we could have. In the days of virtualization, this becomes 
easier and cheaper, so you should really investigate why you should not 
have a "play and break" platform for your mission critical stuff. If 
your managers start dismissing the idea in terms of cost (hardware and 
manhours dedicated to the task), remind them what could they loose if 
they loose everything or something works in degraded mode. If that 
represents an amount of money larger than the cost of running this 
test/assurance platform, they have no case.

As for the vendor testing, RedHat does perform in my experience fairly 
conservative and careful testing. But in order to break something, you 
do need really need to change ABI/API. A device driver or a change of 
kernel parameters can really have an impact on your mission critical 
app. I remember the SCSI affinity patches of the RHEL 3.0 series and how 
they affected read performance, making standard Dell kit (PERC U320 
controllers) run soooo slowly for some workloads. For a while, the 
vendor was arguing that the settings were good for some workloads and at 
the end they removed them I think. My point. Yes, you could tune, yes, 
they might fix it, but if you have a mission critical app and you roll 
the patch, only to find your Oracle running in dog mode thirty minutes 
later, by the time you scratch your head enough to connect it with the 
update, you lost revenue and uptime which would be avoided if you had a 
test platform.

The best assurance is your own testing. Only my honest opinion.

-- 
--
George B. Magklaras

Senior Computer Systems Engineer/UNIX Systems Administrator
The Biotechnology Centre of Oslo,
University of Oslo
http://www.biotek.uio.no/

EMBnet Norway: http://www.biotek.uio.no/EMBNET/




AB wrote:
> Hello all,
> 
> We run several mission critical RHEL 4 AS servers and we are currently having a
> bit of an internal debate regarding the installation of official RedHat updates.
> 
> Several of my colleagues think that installing the RedHat updates is too
> dangerous because it could potentially break another package and/or another
> piece of our or a third party's software.
> 
> These are my arguing points as to why we should apply the updates. Please
> correct me if I have misunderstood any of the points I make, and feel free to
> add more to the list (the more the better): -
> .	All RHEL updates are exhaustively tested by RedHat to make sure they
> will not break other official components of the OS.
> .	RHEL updates are only ever bugfix / security updates. An API/ABI will
> never be changed as part of an update. If one of our programs was compiled
> against a library which later got updated, the program would not be negatively
> affected by the update.
> .	RHEL updates are tested against most of the large certified applications
> (such as Oracle etc.) before release.
> .	RedHat don't release updates just for the fun of it, they release them
> to fix new security holes to prevent our systems been broken into, and to fix
> known critical bugs to keep our systems stable and our data intact.
> 
> On a slightly different note, does anyone know of a certification framework we
> could develop our applications to, to provide the best possible compatibility
> with the underlying RHEL OS?
> 
> What do most other organisations in our position do? I'd be especially
> interested to hear from other companies using RHEL 4 AS who must provide 24/7
> availability, and what RedHat's official line on the matter is.
> 
> 
> Regards,
> Adam
>