[Spacewalk-list] Spacewalk redundancy
Konstantin Raskoshnyi
konrasko at gmail.com
Thu May 26 18:03:08 UTC 2016
Thanks. I was thinking about NFS & Dedicated DB cluster. Will try later
Have a great day
On Thu, May 26, 2016 at 8:19 AM, William H. ten Bensel <WHTENBEN at up.com>
wrote:
> This is a solution (might not be the best way) to prevent re-registering,
> hours of importing/exporting channels, and setting up an entire new
> spacewalk server with lots of downtime. Can be used in case the Active
> Spacewalk needs to be down longer then the Service level agreement. For
> this setup, the recovery is very quick as long as someone or automated
> scripts have access to start/stop spacewalk, update DNS for the cname flip,
> and switch out the storage. However there is some extra cost and some
> additional man hours on initial setup. But if you need 99.9+ SLA, this
> will work.
>
> Note: Updating /etc/sysconfig/network, /etc/hosts, and hostname for some
> reason is a requirement. Was not able to get spacewalk to function
> properly without doing these. If someone knows how to get around updating
> the files/hostname and tell Spacewalk to go off of what DNS returns, that
> would be great!
>
> Server Setup (Oracle DB off host. Uncertain of how to do postgress and On
> host DB, but should be able to use the same logic.)
>
> cname (That is fully qualified ie. shortname.domain) pointed to 1
> Active Spacewalk server.
>
> 1 Active Spacewalk Server "cname -> hostname.domain (A record)"
> 1 Hot Standby Server "hostname.domain (A record)"
>
> Create Repo/snapshot of the Spacewalk and Jpackage Repos with the
> current version that are installed on Act/Standby server.
> Pros of creating repo/snapshot:
> Guarantee version does not change.
> The repos above provide a quicker recovery if both servers
> go down.
> Download once and use local install
> Backup of the entire Spacewalk server using company standard
> backup policies/methodology.
>
>
> Spacewalk setup on Active/Standby:
>
> Note: If upgrading, need to do both Active/Standby at the same
> time. Just make sure you only run the schema update once!!!
>
> 1. The following file systems on NFS/SAN:
> Notes: You can reduce the # of mounts by overriding the
> defaults in /etc/rhn/rhn.conf. Defaults can be found in
> /usr/share/rhn/config-defaults/*.conf.
> There is a benefit to stay with the defaults as to reduce
> conf changes.
> All of these mounts may not be needed, but these have been
> proven to work.
> If you already have one setup, you can stop the spacewalk
> and move the files/directories to the NFS/SAN.
>
> Someone suggested to ln -s all of these to on NFS mount
> and then create directories in that mount.... That is an option as well....
> imo, not a good one. The less links, the better.
> If choosing this method... ensure that the link ownerships
> are correct... ie chown -h jabber:jabber /etc/jabberd.
>
> xT/yG /var/satellite (Primary is xT, Standby is yG) ->
> This is because Primary will replace Standby)
> 1G /config
> 1G /etc/cobbler
> 1G /etc/jabberd (PERMISSIONS MUST BE: jabber:jabber or
> else osa-dispatcher will not start)
> 1G /etc/httpd
> 1G /etc/rhn
> 1G /etc/pki/spacewalk
> 1G /usr/share/rhn
> 1G /usr/share/spacewalk
> 30G/yG /var/cache/rhn
> 1G /var/lib/cobbler
> 2G /var/lib/rhn
> 1G /var/log/cobbler
> 1G /var/log/rhn
> 2G /var/log/httpd
> 10G/yG /var/www
> xG/yG /somemount/ (configs, custom built scripts, logging,
> start/stop scripts, etc)
>
> 2. Setup proxy
> echo "export http_proxy=xxxxxx:port#" >
> /etc/profile.d/httpproxy.sh
> echo "export http_proxy=xxxxxx:port#" >
> /etc/profile.d/httpproxy.csh
>
> 3. Oracle client install (Do not use postgres, so someone will
> have to provide guidance on that)
> yum install compat-gcc-34 compat-gcc-34-c++ gcc-c++ gdb
> unixODBC-devel glibc-devel libXp xorg-x11-utils xorg-x11-xinit
> glibc-devel.i686 libXp.i686 –y
> https://fedorahosted.org/spacewalk/wiki/FullOracleSetup
> NOTE: Need the DEVEL package as well for Python scripts
> cx_oracle.
>
> 4. Server configuration for the install:
> Update /etc/hosts line with the 'IP cname shortname',
> above "IP hostname.domain hostname".
> Update /etc/sysconfig/network with 'HOSTNAME=cname'
> Update hostname. 'hostname cname'
>
> 5. Install the same version of Spacewalk on both servers using the
> snapshot repo.
>
> 6. Configuration changes/updates.
>
> /etc/yum.conf (This is optional. Do this to prevent
> accidental updates)
> reposdir=/etc/yum.repos.d/MYDIR
> /etc/rhn/rhn.conf
> mv to /somemount/configs/
> ln -s /somemount/configs/rhn.conf /etc/rhn/rhn.conf
> chown -h root:apache /etc/rhn/rhn.conf
>
> edit of /etc/rhn/rhn.conf
> # EZconnect: HOST should be the cname of the DB
> server. Global name should be provided by DBA.
> db_name=//<HOST>/<GLOBAL NAME>
> server.satellite.http_proxy = xxxx:port#
> # The following overrides the defaults in
> # /usr/share/rhn/config-defaults/*.conf
> product_name = xxxxxx
> #OPTIONAL
> java.custom_header = xxxxxx
> #OPTIONAL
> java.config_file_edit_size = 128 #To
> support larger config files
> web.default_mail_from = xxxxxx <yyyyy at cname>
> #Uniquely identify spacewalks in email
> web.session_cookie_lifetime = ###
> #OPTIONAL, used to extend session timeouts
> web.session_database_lifetime = ###
> #OPTIONAL, used to extend session timeouts
>
> # To support larger tasks and reposyncs.
> # Increase java heap size and logging:
> https://access.redhat.com/solutions/43122
> taskomatic.java.maxmemory=### >= 4096
> taskomatic.java.initmemory=### > 256 (default is
> 256)
> taskomatic.logfile.maxsize=###m
> java.taskomatic_channel_repodata_workers=# >2
> (default is 2)
> #taskomatic.logfile.loglevel = DEBUG
>
> # Increase rhn-search
> https://access.redhat.com/solutions/322093
> rhn-search.java.maxmemory = 768
>
>
> /var/lib/rhn/rhn-satellite-prep/satellite-local-rules.conf
> serverDOTsatelliteDOThttp_proxy=xxxxxx:port#
>
> /etc/sysconfig/tomcat6
> mv to /somemount/configs/
> ln -s /somemount/configs/tomcat6
> /etc/sysconfig/tomcat6
>
> edit of /etc/sysconfig/tomcat6
> # Increase java heap size:
> https://access.redhat.com/solutions/43122
> -Xms512m
> -Xmx8192m
>
> ln -s /var/www/html/pub/RHN-ORG-TRUSTED-SSL-CERT
> /usr/share/rhn/RHN-ORG-TRUSTED-SSL-CERT
>
> jabberd special validation!!!
> /etc/pki/spacewalk/jabberd/server.pem Ownership
> --> jabber:jabber
> /etc/jabberd/server.pem Ownership
> --> root:jabber
>
> Ensure that these are backed up.
> rhn-org-httpd-ssl-key-pair-cname-1.0-1.noarch
> /root/ssl-build with SSL Certs, .crt files, and
> rpms
> /var/www/html/pub/RHN-ORG-TRUSTED-SSL-CERT
>
> /var/www/html/pub/rhn-org-trusted-ssl-cert-1.0-1.noarch.rpm
>
> Fix /etc/logrotate.d/httpd by adding the following.
> rotate 4
> compress
> dateext
>
>
> Create custom init.d start/stop scripts that do the following.
> Note: Put the start/stop script on the ACTIVE NFS/SAN. In the
> init.d, check to see if they exist. If not do not run, but make sure that
> the spacewalk services are disabled!!!!
>
> Stop:
> Ensure that spacewalk is disabled. Do not want to have 2
> inits calling the same services.
> for spaceservice in `/usr/sbin/spacewalk-service
> list |grep "4:" |awk '{print $1}'`; do
> /usr/sbin/spacewalk-service disable
> $spaceservice
> done
> Validate script running as root
> Validate spacewalk running
> Run spacewalk-service stop
>
> Remove cname from /etc/sysconfig/network, /etc/hosts,
> reset hostname to hostname.domain
> Ensure that kill any/all scripts that are running
> Ensure that any scripts will not start/kickoff. ie lock
> file or all scripts verify that the spacewalk is up (or both)
>
> Start:
> Ensure that spacewalk is disabled. Do not want to have 2
> inits calling the same services.
> for spaceservice in `/usr/sbin/spacewalk-service
> list |grep "4:" |awk '{print $1}'`; do
> /usr/sbin/spacewalk-service disable
> $spaceservice
> done
> Validate script running as root
> Validate server has cname.
> IF not, run the stop script. (Which should ensure
> that the files and hostname are set correctly)
> exit
>
> Validate spacewalk is not running.
> If it is, consider it a restart
> Run spacewalk-service restart
> If not.
> Update cname in /etc/sysconfig/network,
> /etc/hosts, reset hostname to cname
> Run spacewalk-service start
>
> Validate all service started. If not send notification.
>
>
>
> Recovery/Failover/DR/Migration:
>
> Standby server:
> Unmount all mounts. ### Comment out in /etc/fstab
> Active server:
> Stop
> Unmount ### Comment out in /etc/fstab so you can update or
> reboot.
> Standby server
> Flip cname to Standby
> Mount all Active Mounts, update /etc/fstab to withstand
> server reboot
> Start
>
> - Thanks and good luck
>
>
>
> From: Konstantin Raskoshnyi <konrasko at gmail.com>
> To: spacewalk-list at redhat.com
> Date: 05/25/2016 04:10 PM
> Subject: [Spacewalk-list] Spacewalk redundancy
> Sent by: spacewalk-list-bounces at redhat.com
> ------------------------------
>
>
>
> This email originated from outside of the company. Please use discretion
> if opening attachments or clicking on links.
> ------------------------------
>
>
> Hello everyone,
> As I see all systems rely on sp server in fact.
> In case of emergency and if sp is dead you'll lose control over your boxes
> and have to install a new instance of sp and re-register machines.
>
> Any solutions in this case?
>
> Thanks This email originated from outside of the company. Please use
> discretion if opening attachments or clicking on links.
>
> _______________________________________________
> Spacewalk-list mailing list
> Spacewalk-list at redhat.com
> https://www.redhat.com/mailman/listinfo/spacewalk-list
>
>
>
> **
>
>
>
> This email and any attachments may contain information that is
> confidential and/or privileged for the sole use of the intended recipient.
> Any use, review, disclosure, copying, distribution or reliance by others,
> and any forwarding of this email or its contents, without the express
> permission of the sender is strictly prohibited by law. If you are not the
> intended recipient, please contact the sender immediately, delete the
> e-mail and destroy all copies.
>
> **
>
>
> _______________________________________________
> Spacewalk-list mailing list
> Spacewalk-list at redhat.com
> https://www.redhat.com/mailman/listinfo/spacewalk-list
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://listman.redhat.com/archives/spacewalk-list/attachments/20160526/0fc2a55f/attachment.htm>
More information about the Spacewalk-list
mailing list