[katello-devel] I had a dream: announcing RPM-based development setup
Lukas Zapletal
lzap at redhat.com
Mon Dec 3 09:41:40 UTC 2012
I am fine with re-ordering the page. Whatever works for managers :-)
LZ
On Fri, Nov 30, 2012 at 12:27:01PM -0500, Bryan Kearney wrote:
> On 11/29/2012 06:19 AM, Lukas Zapletal wrote:
> >Hi all,
> >
> >since Mirek packaged all our development gems as RPMs, it was one simple
> >step to finalize this and offer instructions about how to put production
> >instance into development mode very easily.
> >
> >Today I want to push one another improvement - introduction of bundle.d
> >directory (inspiration from Foreman). If you like it, instructions will
> >be even more simple (and it will be first step towards modularizing
> >Katello).
> >
> >Note I am not trying to replace your development setups, this can work
> >well with git checkouts either on the installed machine or on your
> >laptops (you can configure it to connect to the machine). I would say
> >this setup is complementary, but I would like to present it as the
> >recommended setup for the community, because it's very easy to setup and
> >it just works. People can start hacking katello right away, after few
> >issued commands.
> >
> >The development setup was put alive with the bundler_ext change - I will
> >blog about it later this week.
> >
> >https://fedorahosted.org/katello/wiki/DevelopmentSetup#RPM-basedsetup
> >
> >"""
> >This setup installs on top of production mode adding development gems. You can
> >enable them and turn Katello into development mode and enjoy all the
> >development mode features like autoloading or logging while maintaining the
> >environment. Development setup can be used in conjunction with a git checkout
> >either on the machine or on developers laptop. The usual practice is to develop
> >on a laptop and then using scp/ftp/rsync to overwrite files in
> >/usr/share/katello testing the features on the target environment.
> >
> >This development setup runs under bundler_ext just like production mode does.
> >In this mode, Gemfile file is renamed to Gemfile.in and Katello loads all
> >dependencies using bundler_ext library rather than bundler. It requires all
> >necessary rubygems from the system ignoring versions specified in the Gemfilein
> >and if a dependency is missing, it fails. There is no "bundle install" command
> >avaiable in this mode and bundler_ext can be put in NOSTRICT mode where it
> >ignores missing dependencies (logging error messages) which can be useful for
> >development or installation phase (database migration or seed).
> >
> >This setup works on supported platforms ({{{Fedora/Red Hat and clones}}}) and
> >it's the officially maintained by Katello developers - that means we will make
> >sure it is always working and we will be able to help you with issues on lists
> >and IRC channels.
> >
> >You need to [wiki:Install install Katello first], please check requirements
> >sections and make sure installation is working properly. If you prefer fully
> >automatic installation, just run the following on a clean machine:
> >
> >{{{
> >git clone git://github.com/lzap/ciauciau.git
> >echo 'PATH=$PATH:ciauciau/bin' >> ~/.bashrc
> >source ~/.bashrc
> >ciau -p kt
> >}}}
> >
> >This gives you nightly production installation. Now, you want to enable
> >development gems and put Katello into development mode:
> >
> >{{{
> >yum -y install katello-devel-all
> >}}}
> >
> >Hint: If you only want development gems, install {{{katello-devel}}}, if you
> >only need testing gems install {{{katello-devel-test}}}. Possible packages are:
> >
> > * katello-devel-all (installs all groups bellow)
> > * katello-devel (development rubygem group)
> > * katello-devel-test (unit testing rubygem group)
> > * katello-devel-profiling (profiling rubygem group)
> > * katello-devel-jshintrb (javascript analysis rubygem group)
> >
> >Warning: There is a missing dependency in the katello-devel-jshintrb
> >metapackage. It's recommended to install without this package: {{{yum -y
> >install katello-devel katello-devel-test katello-devel-profiling}}} until we
> >resolve the issue. You will only miss javascript analysis tool which is not
> >needed for development.
> >
> >Uncomment, edit or add the following lines in {{{/etc/sysconfig/katello}}}:
> >
> >{{{
> >KATELLO_ENV=development
> >KATELLO_LOGGING=debug
> >BUNDLER_EXT_GROUPS="development debugging test"
> >BUNDLER_EXT_NOSTRICT=1
> >}}}
> >
> >Explanation of the variables:
> >
> > * KATELLO_ENV - Sets RAILS_ENV variable, basically you turning on autoloading
> >and other features of the development mode of Katello.
> > * KATELLO_LOGGING - Turns on more verbose logging. Watch out because this is
> >very verbose and can load your instance quickly. There are also other logging
> >options in the file (SQL logging for instance).
> > * BUNDLER_EXT_GROUPS - List of space separated bundler groups you want to
> >load. Usually you only need "development test" there. Open
> >/usr/share/katello/Gemfile.in to see all groups.
> > * BUNDLER_EXT_NOSTRICT=1 - Puts bundler_ext in the NOSTRICT mode, that means
> >if it miss a dependency, it will still continue starting up the server. This
> >can be very useful when you miss a dependency (e.g. package does not exist yet)
> >or you want to test some rubygem on your own.
> >
> >You may want to review other parameters in that file and also turn off firewall
> >and SELinux (this is optional but recommended):
> >
> >{{{
> >setenforce 0
> >service iptables stop
> >}}}
> >
> >You can restart katello service and enjoy fully development mode (e.g.
> >autoloading). Many developers use their IDE via SCP/FTP or rsyncing files or
> >simply editing via ssh.
> >
> >{{{
> >service katello restart
> >}}}
> >
> >Warning: It is not recommended to start the katello server on the foreground as
> >root, because this will likely break your installation (pid/temp files will be
> >created as root). Use startup script and tail appropriate files at will. Also,
> >if you overwrite many files, make sure you check the installation before
> >upgrade if you want to return to the RPM version: {{{rpm -qa | grep katello |
> >xargs rpm -V}}}.
> >
> >Advantages of the RPM based development setup are:
> >
> > * Easy automation - deployment can be as short as 10 minutes with snap-guest.
> > * It's close to the production environment.
> > * You can switch back to production.
> > * Possibility to overwrite with official builds from RPMs.
> > * Possibility to upgrade to other version (next day nightly).
> > * You can test SELinux.
> > * Easy to understand and maintain.
> > * Many of the Katello team are using this - easier to find help.
> >
> I totally dig the ciauciau stuff. However... assume I am a stupid
> manager type. I get to the first Q&A about checking out from git,
> and you say "Do the needful". This page is about developer setup,
> so we should be explicit. If i go look at the rest of hte page for
> git I go to the section titled "Set Up Development (using Git) " and
> it assuming a slightly different model. What would you think of:
>
> 1) Kill the sections titled:
>
> Not-Quite-So-Dirt-Simple-But-Still-Easy Approach
> Build Katello from Source
>
> 2) GIve complete steps for the git checkout. this may be editing the
> section (using Git) from above.
>
> My success criteria would be me doing an install, editing a page,
> and seeing it work.
>
> - bk
>
> Fedora Hosted
> Search: [ ] [Search]
> • Fedora Account Login
> • Help/Guide
> • About Trac
> • Fedora Account Sign Up
> • Preferences
>
> • Wiki
> • Timeline
> • Roadmap
> • Browse Source
> • View Tickets
> • Search
>
> wiki:DevelopmentSetup
>
> Context Navigation
>
> • Start Page
> • Index
> • History
>
> ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
> Last modified 5 hours ago Last modified on 11/30/12 11:58:15
>
> 1. Installation Instructions
> 1. RPM-based setup
> 1. Questions and answers
> 2. Dirt Simple Approach
> 3. Not-Quite-So-Dirt-Simple-But-Still-Easy Approach
> 1. Yum Repo Files
> 1. Fedora 16
> 2. Red Hat Enterprise Linux 6
> 2. Install the RPMs
> 3. If setting up for development, stop the production Katello services
> 4. Set Up Development (using Git)
> 5. Build Katello from Source
> 4. Troubleshooting
> 1. Bundle Install error - xyz is missing
> 2. Role "katello" does not exists
> 3. Candlepin Issues
>
> Installation Instructions
>
> • Dirt Simple Approach
> • Not-Quite-So-Dirt-Simple-But-Still-Easy Approach
> • Build Katello from Source
> • Troubleshooting
>
> These instructions were written with a minimum installation, so things might be
> a little verbose. The following instruction have been codified in a set of
> puppet modules. If you would like to use these to gets a new Katello machine
> which "Just Works (tm)" you can follow the steps at Install. If not, follow the
> steps below.
>
> RPM-based setup
>
> This setup installs on top of production mode adding development gems. You can
> enable them and turn Katello into development mode and enjoy all the
> development mode features like autoloading or logging while maintaining your OS
> environment (works on all supported platforms). Development setup can be used
> in conjunction with a git checkout either on the machine or on developer's
> laptop. The usual practice is to develop on a laptop and then using scp/ftp/
> rsync to overwrite files in /usr/share/katello testing the features on the
> target environment. More about this later.
>
> This development setup runs under bundler_ext extension just like production
> mode does. In this mode, Gemfile file distributed as Gemfile.in and Katello
> loads all dependencies using bundler_ext library rather than Bundler. It
> requires all necessary rubygems from the system ignoring versions specified in
> the Gemfile.in and if a dependency is missing, it fails. There is no "bundle
> install" command necessary in this mode and bundler_ext can be put in no-strict
> mode where it ignores missing dependencies (writing error messages on the
> standard error output) which can be useful for development (you want to quickly
> try something on a broken installation) or during installation phase (database
> migration or seed will not fail if an unimportant dependency is missing).
>
> This setup works on supported platforms (Fedora/Red Hat and clones) and it's
> the officially maintained by Katello developers - that means we will make sure
> it is always working and we will be able to help you with issues on lists and
> IRC channels. Also note there are various other guides to setup Katello for
> development on this wiki page, you can use them as a complementary setup on a
> host that was put to development mode using this technique.
>
> You need to install Katello first, please check requirements sections and make
> sure installation is working properly. If you prefer fully automatic
> installation of our nightly build, just run the following on a clean machine:
>
> git clone git://github.com/lzap/ciauciau.git
> echo 'PATH=$PATH:ciauciau/bin' >> ~/.bashrc
> source ~/.bashrc
> ciau -p kt
>
> There are more options for you to install particular Katello version or
> Headpin, see https://github.com/lzap/ciauciau. Now, you want to enable
> development gems and put Katello into development mode:
>
> yum -y install katello-devel-all
>
> Hint: If you only want development gems, install katello-devel, if you only
> need testing gems install katello-devel-test. and so on. Possible rubygem group
> packages are:
>
> • katello-devel-all (installs all groups bellow)
> • katello-devel (development rubygem group)
> • katello-devel-test (unit testing rubygem group)
> • katello-devel-profiling (profiling rubygem group)
> • katello-devel-jshintrb (javascript analysis rubygem group)
>
> Now you want to put Katello into development mode. Uncomment, edit or add the
> following lines in /etc/sysconfig/katello:
>
> KATELLO_ENV=development
> KATELLO_LOGGING=debug
> BUNDLER_EXT_GROUPS="development debugging test"
> BUNDLER_EXT_NOSTRICT=1
>
> Explanation of the variables:
>
> • KATELLO_ENV - Sets RAILS_ENV variable, basically it is turning on
> autoloading and other features of the development mode of Rails.
> • KATELLO_LOGGING - Turns on more verbose logging. Watch out because logging
> in debug level can be very verbose and fill your instance quickly with
> hundreds of megabytes. There are also other logging options in the file
> (SQL logging for instance).
> • BUNDLER_EXT_GROUPS - List of space separated bundler groups you want to
> load. Usually you only need "development test" there. Open /usr/share/
> katello/Gemfile.in to see all groups.
> • BUNDLER_EXT_NOSTRICT=1 - Puts bundler_ext in the NOSTRICT mode, that means
> if it miss a dependency, it will still continue starting up the server.
> This can be very useful when you miss a dependency (e.g. package does not
> exist yet) or you want to test some rubygem on your own.
>
> You may want to review other parameters in that file and also turn off firewall
> and SELinux (this is optional but if you are going to create new files or
> overwrite installation with your own version you will likely need at least
> SELinux to be turned off):
>
> setenforce 0
> service iptables stop
>
> You can restart katello service and enjoy fully development mode (e.g.
> autoloading). Many developers use their IDE via SCP/FTP or rsyncing files or
> simply editing via ssh using Vim or Emacs.
>
> service katello restart
>
> Warning: It is not recommended to start the katello server on the foreground as
> root using rails server, because this will likely break your installation (pid/
> temp files will overwritten with incorrect root permissions). Use startup
> script and tail appropriate files at will.
>
> If you overwrite some installed files, make sure you check the installation
> before upgrade if you want to return to the RPM version: rpm -qa | grep katello
> | xargs rpm -V. Please note upgrades are not supported nightly to nightly, only
> latest stable version to nightly. Therefore it is recommended to automate your
> development setup installation so you can quickly re-create your environment
> without manual intervention.
>
> Advantages of the RPM based development setup are:
>
> • Easy automation - deployment can be as short as 10 minutes with snap-guest.
> • It's close to the production environment - you can "hack" features or fix
> bugs easily on a given production setup with a reproducer.
> • You can switch back to production - maybe you want to test your patch under
> higher load.
> • Possibility to overwrite with official builds from RPMs - if you overwrite
> some files, with some care you can put everything back as it was installed
> from RPM.
> • Possibility to upgrade to other version - you can upgrade your instance,
> but note this is not supported (usually works).
> • You can test SELinux - maintaining our SELinux policy can be challenging
> and it's great to have the option to debug and to hack installation while
> SELinux is fully enabled.
> • Easy to understand and maintain - just few commands to remember.
> • Many of the Katello team are using this - easier to find help and support.
>
> Questions and answers
>
> Can I checkout our git?
>
> Sure, you can checkout it in a separate directory, then stop katello instance
> and start it from your directory using bundler (downloading dependencies from
> rubygems.org). All services are running and has been initialized as well as the
> katello database and /etc/katello/katello.yml is also configured correctly,
> there are no changes necessary.
>
> Can I overwrite files in /usr/share/katello?
>
> You can, actually development mode allows you to change files and Rails server
> will automatically load the changes on the fly. You can also rename this
> directory and checkout git there, although this is not recommended. You will
> need to turn off SELinux or properly relabel the files after this change. If
> you need to checkout git, stop katello instance, checkout in a different
> directory and start your own instance using bundler.
>
> How to connect my XYZ IDE?
>
> If you need to use your IDE, you have several options. You can either mount the
> /usr/share/katello somehow, or you can rsynch the changed files there. In both
> cases you need to turn off selinux, or make sure files has correct contexts.
>
> But the preferred way is to run katello locally and configure it to connect to
> the backend engines on the host as well as katello database. In this case, you
> need to turn off iptables and also reconfigure postgres to listen on all
> interfaces: https://github.com/lzap/ciauciau/blob/master/steps/
> 499-pg-accept-all.sh In this configuration, you don't even need to install
> development dependencies or enable it in the sysconfig file, but it's good to
> have it.
>
> Dirt Simple Approach
>
> In case you did not read above, see Install for the dirt simple approach.
>
> Not-Quite-So-Dirt-Simple-But-Still-Easy Approach
>
> This approach assumes you will be running Katello, Candlepin, and Pulp all
> together on a single machine.
>
> First things first:
>
> setenforce 0
> Change to read as permissive - to keep over reboots within configuration file - /etc/selinux/config
> SELINUX=permissive
> yum install wget
>
> Red Hat Enterprise Linux 6
>
> Katello requires the entablement of the [rhel-6-server-optional-rpms] repo.
> - If you are using subscription-manager this is found in your /etc/yum.repos.d/redhat.repo file.
> Set this repo to: enabled = 1
> - If you are using yum-rhn-plugin this can be enabled using
> rhn-channel --add --channel=rhel-x86_64-server-optional-6
>
>
> Install EPEL for RHEL 6
>
> $ rpm -Uvh http://download.fedora.redhat.com/pub/epel/6/x86_64/epel-release-6-5.noarch.rpm
>
> Yum Repo Files
>
> You'll need yum repo files for Candlepin, Pulp, and Katello. Those can be
> obtained by installing the katello-repos-latest rpm. For example,
>
> Fedora 16
>
> rpm -Uvh http://fedorapeople.org/groups/katello/releases/yum/1.0/Fedora/16/x86_64/katello-repos-latest.rpm
>
> Red Hat Enterprise Linux 6
>
> rpm -Uvh http://fedorapeople.org/groups/katello/releases/yum/1.0/RHEL/6/x86_64/katello-repos-latest.rpm
>
> Install the RPMs
>
> Now with the correct repo files we will install everything in one go:
>
> yum install katello-all
>
> Once installation is complete, everything will need to be configured. To do so,
> please run:
>
> sudo katello-configure
>
> katello-configure can take two or three minutes to complete. If everything
> completes without error messages, then Katello, along with Candlepin and Pulp,
> was configured correctly. You may access your new running instance of Katello
> at https://<hostname>/katello.
>
> These steps get you a running instance of Katello which is good for testing the
> RPMs, etc.
>
> if you want to be able to develop Katello, please continue
>
> If setting up for development, stop the production Katello services
>
> Stop the Katello services
>
> service katello-jobs stop && service katello stop
>
> Disable them so they won't start up after a reboot
>
> systemctl disable katello-jobs.service && systemctl disable katello.service
>
> Set Up Development (using Git)
>
> First install Git on your system if it is not already installed
>
> yum install git
> git clone --recursive <your_forked_github_url>
>
> On GitHub, fork the Katello project and submit changes via pull requests.
>
> Once you have a GitHub account, have forked katello, and have a Git repo on
> your machine you can then disable the RPM based instances of Katello and then
> run Katello from a Rails thin server from within your git checkout.
>
> There's a few more development libraries that will be needed:
>
> yum groupinstall "Development Tools"
> yum install ruby-devel postgresql-devel sqlite-devel libxml2 libxml2-devel libxslt libxslt-devel
>
> Make sure your bundle is complete
>
> cd ~/path/to/katello/git/checkout/src/
> mv /etc/katello/katello.yml /etc/katello/katello.yml.rpm_based
> ln -s ~/path/to/katello/git/checkout/src/config/katello.yml /etc/katello/katello.yml
>
> Install the required Rubygems via Bundler:
>
> # on Fedora:
> yum install katello-devel-all
> # everywhere else
> bundle install
>
> Reset all the DBs to run with katello in your git checkout:
>
> sudo ./script/reset-oath
> ./script/katello-reset-dbs development ~/path/to/katello/git/checkout/src/
>
> Run Katello from the thin server in your git checkout's src directory
>
> cd ~/path/to/katello/git/checkout/src/
> export RAILS_RELATIVE_URL_ROOT=/katello
> ./script/rails s thin
>
> If RAILS_RELATIVE_URL_ROOT is not exported, then the Rails server gets stuck
> and will not answer URL requests.
>
> Your development Katello server is now accessible at http://<host_name>:3000/
> katello
>
> Run your katello-jobs in dev mode as well:
>
> cd ~/path/to/katello/git/checkout/src/
> ./script/delayed_job start
>
> Build Katello from Source
>
> In order to build Katello from source, steps must be taken to account for the
> inclusion of the ConvergeUI git submodule. For the steps below, the assumption
> is that you have cloned the Katello source as described above (and that the git
> submodule is checked out to the currently pointed-to hash). Also, see
> developing with ConvergeUI.
>
> Starting from the base directory, build and install ConvergeUI from HEAD:
>
> cd src/vendor/converge-ui/
> tito build --test --rpm
> rpm -Uvh /tmp/tito/noarch/converge-ui-devel-*.rpm
>
> Now, change directories back to katello/src and build Katello from HEAD:
>
> cd ../../
> tito build --test --rpm
>
> Troubleshooting
>
> Bundle Install error - xyz is missing
>
> If you see this message, check the git setup section above. You are missing
> some dependnecies, most likely "*-devel." rpms.
>
> Role "katello" does not exists
>
> If Katello yells FATAL: role "katello" does not exist, then that means that the
> postgres "katello" user has not been created. Run:
>
> sudo su - postgres -c 'createuser -dls katello'
>
> Candlepin Issues
>
> For Candlepin related errors, please see the Candlepin troubleshooting section
>
> Download in other formats:
>
> • Plain Text
>
> ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
> Trac Powered
>
> Powered by Trac 0.12.4
> By Edgewall Software.
>
> Gravatar support by Tracvatar 1.7
>
> Visit the Trac open source project at
> http://trac.edgewall.org/
>
> SECURITY NOTE: file ~/.netrc must not be accessible by others
> _______________________________________________
> katello-devel mailing list
> katello-devel at redhat.com
> https://www.redhat.com/mailman/listinfo/katello-devel
--
Later,
Lukas "lzap" Zapletal
#katello #systemengine
More information about the katello-devel
mailing list