[katello-devel] Bundler vs rpm-gems - in more detail

Tue Aug 28 07:21:02 UTC 2012

On 23.08.12 14:42, Dmitri Dolguikh wrote:
> On 23/08/12 12:56 PM, Petr Chalupa wrote:
>> My original email raised a lot of questions, I should wrote it with
>> more details. I apology for that. I'll try again and elaborate.
>>
>> = Goals
>>
>> 1) To be able to implement the solution in a week or so.
>> 2) Not to affect deployment, keep it same way as now.
>> 3) When you are installing development environment on any rpm-based
>> system rpm-gems are used even if they are older then gems in our
>> gem-repo. (this is borken right now)
>> 4) When you are installing development environment in any non-rpm-base
>> system you get installed gems as close as possible to what would you
>> get on fedora 16. (you cannot install with bundle install only right now)
>>   * (optionally including the security patches, lets discuss this later)
>>
>> = What was discussed, why I think it wont work
>>
>> == Branch for each platform
>>
>> * There would be little difference between the branches (only Gemfile
>> or Gemfile.lock). Rest of the code will be same.
>> * It would be too much work to maintain it.
>>
>> == Gemfile.lock per platform and environment
>>
>> * We would have to have 8 Gemfile.locks and maintain them all (4
>> platforms multiplied by two environments)
>> * Too much work unless its automated.
>> * Automation would take time to setup.
>>
>> == Aeolus solution, bundler_ext [2]
>>
>> * Dependencies resolution is unaffected so 'bundle install' cannot be
>> used to install development dependencies, it would ignore rpm-gems and
>> install gems from gem-repo.
>> * If bundler_ext is used for setup development environment, control
>> over which versions are required is lost (requiring is not done by
>> bundler)
>> * It affects only how requirements are loaded [1], it's useful only
>> for deployment.
>> * (We could use it in deployment to get rid of bundler, but that is
>> out of this discussion's scope see Goal.2)
>>
>> == bundle install --without default
>>
>> This installs only development gems, but:
>>
>> * If some of the development gems has a production gem as dependency
>> (eg. ActiveSupport), the gem is reinstalled by bundler from a gem-repo
>> over the older rpm-gem.
>> * If only development gems are installed, bundler cannot be used to
>> require gems. It will raise LoadError because from bundler's point of
>> view production gems are not installed.
>> * It would have to be combined with bundler_ext to find production
>> rpm-gems, so you would loose control over which version are loaded again
>> * If you forget to add '--without default' when you run 'bundle
>> install' unwanted versions are installed.
>>
>> == vendor/cache
>>
>> Move all gems to src/vendor/cache (directly into katello/master or as
>> a git sub-module) and remove sources form Gemfile completely. 'bundle
>> install' installs gems form vendor/cache automatically if there are no
>> sources in Gemfile.
>>
>> * to have .gem files directly in katello/master is ugly
>> * submodules are real pain
> no need to use sub-modules
>> * git is not that good with binary data
>>
> gems are mostly ruby, very not a lot of binary data.

gems would be stored in vendor/cache as .gem files, one file per gem 
which are actually tar.gz (binary). That is what I meant sorry about the 
confusion.

>> = My proposal
>>
>> == Bundler patch
>>
>> * it would be loaded in Gemfile, so every tool using bundler will get it
> I don't think a single Gemfile is going to solve the issue: we'll need
> to relax version constraints in Gemfile, and add a bunch of ruby-version
> specific gems...
>>
>> * it would ensure that if rpm-gem is present it's used even if there
>> is newer version available in gem-repo
>> * ensure that locally installed gems are not confused with rpm-gems
> You didn't explain how you are planning to differentiate between the two
> - system-wide bundler-/ruby-gem-installed gems and rpm-installed gems...

I don't think this is a problem, see [1] how I did it.

>>
>> * patch is optional, can be turned on/off by ENV variable or by options
>> * (I am also considering to add optional check that all production
>> gems are installed as rpms)
>>
>> Deployment is unaffected, all dependencies are satisfied by rpm-gems.
>>
>> When you run bundle install to get development environment on any
>> rpm-based machine, all your rpms have priority and they are used. All
>> missing development gems are installed from our gem-repo.
> I think this is the crux of the issue. Once we have a custom gem
> repository, the above step has very little practical value. A ruby gem
> is a ruby gem no matter what place in the system it's installed in.

IMO This has a big practical value. When you are fixing a bug you want 
to be sure that you have same rpm-gems installed and required as in 
production where the bug was discovered.

>>
>> On non-rpm-based machine, 'bundle install' installs all gems
>> production and development from our gem-repo.
>>
>> == Gem Repository
>>
>> Bundler patch will work only if all development and production gems
>> are in the gem-repo. To get it fixed quickly 'bundle package' can be
>> used to get all the gems. Missing gems can be then uploaded to
>> katello-thirdparty (source for our gem-repo)
>>
>> = Summary
>>
>> I think that Bundler patch and updating gem-repo to contain all gems
>> is the only one which can be done fast and will work on any machine in
>> deployment and development. I tried to be thorough but if I missed
>> something or if you have better ideas please let me know.
> The issue with dependency version variability needs solving. The patch
> itself has little practical value, bundle install from custom repository
> is sufficient. The irony is that you are trying to solve the problem
> Bundler has successfully solved awhile ago. I'm baffled as to why we
> continue to insist on solving the problems the "katello" way.
>
> -d
>
>>
>> Petr
>>
>> [1]
>> https://github.com/aeolusproject/conductor/blob/master/src/config/application.rb#L21
>>
>> [2] https://github.com/aeolus-incubator/bundler_ext
>>
>> _______________________________________________
>> katello-devel mailing list
>> katello-devel at redhat.com
>> https://www.redhat.com/mailman/listinfo/katello-devel
>
>
> _______________________________________________
> katello-devel mailing list
> katello-devel at redhat.com
> https://www.redhat.com/mailman/listinfo/katello-devel

[1] https://github.com/Katello/katello/pull/526