[Spacewalk-list] Spacewalk redundancy
William H. ten Bensel
WHTENBEN at up.com
Thu May 26 15:19:53 UTC 2016
This is a solution (might not be the best way) to prevent re-registering,
hours of importing/exporting channels, and setting up an entire new
spacewalk server with lots of downtime. Can be used in case the Active
Spacewalk needs to be down longer then the Service level agreement. For
this setup, the recovery is very quick as long as someone or automated
scripts have access to start/stop spacewalk, update DNS for the cname
flip, and switch out the storage. However there is some extra cost and
some additional man hours on initial setup. But if you need 99.9+ SLA,
this will work.
Note: Updating /etc/sysconfig/network, /etc/hosts, and hostname for some
reason is a requirement. Was not able to get spacewalk to function
properly without doing these. If someone knows how to get around updating
the files/hostname and tell Spacewalk to go off of what DNS returns, that
would be great!
Server Setup (Oracle DB off host. Uncertain of how to do postgress and On
host DB, but should be able to use the same logic.)
cname (That is fully qualified ie. shortname.domain) pointed to 1
Active Spacewalk server.
1 Active Spacewalk Server "cname -> hostname.domain (A record)"
1 Hot Standby Server "hostname.domain (A record)"
Create Repo/snapshot of the Spacewalk and Jpackage Repos with the
current version that are installed on Act/Standby server.
Pros of creating repo/snapshot:
Guarantee version does not change.
The repos above provide a quicker recovery if both servers
go down.
Download once and use local install
Backup of the entire Spacewalk server using company standard
backup policies/methodology.
Spacewalk setup on Active/Standby:
Note: If upgrading, need to do both Active/Standby at the same
time. Just make sure you only run the schema update once!!!
1. The following file systems on NFS/SAN:
Notes: You can reduce the # of mounts by overriding the defaults
in /etc/rhn/rhn.conf. Defaults can be found in
/usr/share/rhn/config-defaults/*.conf.
There is a benefit to stay with the defaults as to reduce
conf changes.
All of these mounts may not be needed, but these have been
proven to work.
If you already have one setup, you can stop the spacewalk
and move the files/directories to the NFS/SAN.
Someone suggested to ln -s all of these to on NFS mount
and then create directories in that mount.... That is an option as
well.... imo, not a good one. The less links, the better.
If choosing this method... ensure that the link ownerships
are correct... ie chown -h jabber:jabber /etc/jabberd.
xT/yG /var/satellite (Primary is xT, Standby is yG) -> This
is because Primary will replace Standby)
1G /config
1G /etc/cobbler
1G /etc/jabberd (PERMISSIONS MUST BE: jabber:jabber or
else osa-dispatcher will not start)
1G /etc/httpd
1G /etc/rhn
1G /etc/pki/spacewalk
1G /usr/share/rhn
1G /usr/share/spacewalk
30G/yG /var/cache/rhn
1G /var/lib/cobbler
2G /var/lib/rhn
1G /var/log/cobbler
1G /var/log/rhn
2G /var/log/httpd
10G/yG /var/www
xG/yG /somemount/ (configs, custom built scripts, logging,
start/stop scripts, etc)
2. Setup proxy
echo "export http_proxy=xxxxxx:port#" >
/etc/profile.d/httpproxy.sh
echo "export http_proxy=xxxxxx:port#" >
/etc/profile.d/httpproxy.csh
3. Oracle client install (Do not use postgres, so someone will
have to provide guidance on that)
yum install compat-gcc-34 compat-gcc-34-c++ gcc-c++ gdb
unixODBC-devel glibc-devel libXp xorg-x11-utils xorg-x11-xinit
glibc-devel.i686 libXp.i686 –y
https://fedorahosted.org/spacewalk/wiki/FullOracleSetup
NOTE: Need the DEVEL package as well for Python scripts
cx_oracle.
4. Server configuration for the install:
Update /etc/hosts line with the 'IP cname shortname',
above "IP hostname.domain hostname".
Update /etc/sysconfig/network with 'HOSTNAME=cname'
Update hostname. 'hostname cname'
5. Install the same version of Spacewalk on both servers using the
snapshot repo.
6. Configuration changes/updates.
/etc/yum.conf (This is optional. Do this to prevent
accidental updates)
reposdir=/etc/yum.repos.d/MYDIR
/etc/rhn/rhn.conf
mv to /somemount/configs/
ln -s /somemount/configs/rhn.conf
/etc/rhn/rhn.conf
chown -h root:apache /etc/rhn/rhn.conf
edit of /etc/rhn/rhn.conf
# EZconnect: HOST should be the cname of the DB
server. Global name should be provided by DBA.
db_name=//<HOST>/<GLOBAL NAME>
server.satellite.http_proxy = xxxx:port#
# The following overrides the defaults in
# /usr/share/rhn/config-defaults/*.conf
product_name = xxxxxx #OPTIONAL
java.custom_header = xxxxxx #OPTIONAL
java.config_file_edit_size = 128 #To
support larger config files
web.default_mail_from = xxxxxx <yyyyy at cname>
#Uniquely identify spacewalks in email
web.session_cookie_lifetime = ### #OPTIONAL,
used to extend session timeouts
web.session_database_lifetime = ### #OPTIONAL,
used to extend session timeouts
# To support larger tasks and reposyncs.
# Increase java heap size and logging:
https://access.redhat.com/solutions/43122
taskomatic.java.maxmemory=### >= 4096
taskomatic.java.initmemory=### > 256 (default is
256)
taskomatic.logfile.maxsize=###m
java.taskomatic_channel_repodata_workers=# >2
(default is 2)
#taskomatic.logfile.loglevel = DEBUG
# Increase rhn-search
https://access.redhat.com/solutions/322093
rhn-search.java.maxmemory = 768
/var/lib/rhn/rhn-satellite-prep/satellite-local-rules.conf
serverDOTsatelliteDOThttp_proxy=xxxxxx:port#
/etc/sysconfig/tomcat6
mv to /somemount/configs/
ln -s /somemount/configs/tomcat6
/etc/sysconfig/tomcat6
edit of /etc/sysconfig/tomcat6
# Increase java heap size:
https://access.redhat.com/solutions/43122
-Xms512m
-Xmx8192m
ln -s /var/www/html/pub/RHN-ORG-TRUSTED-SSL-CERT
/usr/share/rhn/RHN-ORG-TRUSTED-SSL-CERT
jabberd special validation!!!
/etc/pki/spacewalk/jabberd/server.pem Ownership
--> jabber:jabber
/etc/jabberd/server.pem Ownership
--> root:jabber
Ensure that these are backed up.
rhn-org-httpd-ssl-key-pair-cname-1.0-1.noarch
/root/ssl-build with SSL Certs, .crt files, and
rpms
/var/www/html/pub/RHN-ORG-TRUSTED-SSL-CERT
/var/www/html/pub/rhn-org-trusted-ssl-cert-1.0-1.noarch.rpm
Fix /etc/logrotate.d/httpd by adding the following.
rotate 4
compress
dateext
Create custom init.d start/stop scripts that do the following.
Note: Put the start/stop script on the ACTIVE NFS/SAN. In the
init.d, check to see if they exist. If not do not run, but make sure that
the spacewalk services are disabled!!!!
Stop:
Ensure that spacewalk is disabled. Do not want to have 2
inits calling the same services.
for spaceservice in `/usr/sbin/spacewalk-service
list |grep "4:" |awk '{print $1}'`; do
/usr/sbin/spacewalk-service disable
$spaceservice
done
Validate script running as root
Validate spacewalk running
Run spacewalk-service stop
Remove cname from /etc/sysconfig/network, /etc/hosts,
reset hostname to hostname.domain
Ensure that kill any/all scripts that are running
Ensure that any scripts will not start/kickoff. ie lock
file or all scripts verify that the spacewalk is up (or both)
Start:
Ensure that spacewalk is disabled. Do not want to have 2
inits calling the same services.
for spaceservice in `/usr/sbin/spacewalk-service
list |grep "4:" |awk '{print $1}'`; do
/usr/sbin/spacewalk-service disable
$spaceservice
done
Validate script running as root
Validate server has cname.
IF not, run the stop script. (Which should ensure
that the files and hostname are set correctly)
exit
Validate spacewalk is not running.
If it is, consider it a restart
Run spacewalk-service restart
If not.
Update cname in /etc/sysconfig/network,
/etc/hosts, reset hostname to cname
Run spacewalk-service start
Validate all service started. If not send notification.
Recovery/Failover/DR/Migration:
Standby server:
Unmount all mounts. ### Comment out in /etc/fstab
Active server:
Stop
Unmount ### Comment out in /etc/fstab so you can update or
reboot.
Standby server
Flip cname to Standby
Mount all Active Mounts, update /etc/fstab to withstand
server reboot
Start
- Thanks and good luck
From: Konstantin Raskoshnyi <konrasko at gmail.com>
To: spacewalk-list at redhat.com
Date: 05/25/2016 04:10 PM
Subject: [Spacewalk-list] Spacewalk redundancy
Sent by: spacewalk-list-bounces at redhat.com
This email originated from outside of the company. Please use discretion
if opening attachments or clicking on links.
Hello everyone,
As I see all systems rely on sp server in fact.
In case of emergency and if sp is dead you'll lose control over your boxes
and have to install a new instance of sp and re-register machines.
Any solutions in this case?
Thanks This email originated from outside of the company. Please use
discretion if opening attachments or clicking on links.
_______________________________________________
Spacewalk-list mailing list
Spacewalk-list at redhat.com
https://www.redhat.com/mailman/listinfo/spacewalk-list
**
This email and any attachments may contain information that is confidential and/or privileged for the sole use of the intended recipient. Any use, review, disclosure, copying, distribution or reliance by others, and any forwarding of this email or its contents, without the express permission of the sender is strictly prohibited by law. If you are not the intended recipient, please contact the sender immediately, delete the e-mail and destroy all copies.
**
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://listman.redhat.com/archives/spacewalk-list/attachments/20160526/50ccb387/attachment.htm>
More information about the Spacewalk-list
mailing list