[katello-devel] Bundler vs rpm-gems - in more detail

Tue Aug 28 09:08:57 UTC 2012

On 28/08/12 08:21 AM, Petr Chalupa wrote:
>
>
> On 23.08.12 14:42, Dmitri Dolguikh wrote:
>> On 23/08/12 12:56 PM, Petr Chalupa wrote:
>>> My original email raised a lot of questions, I should wrote it with
>>> more details. I apology for that. I'll try again and elaborate.
>>>
>>> = Goals
>>>
>>> 1) To be able to implement the solution in a week or so.
>>> 2) Not to affect deployment, keep it same way as now.
>>> 3) When you are installing development environment on any rpm-based
>>> system rpm-gems are used even if they are older then gems in our
>>> gem-repo. (this is borken right now)
>>> 4) When you are installing development environment in any non-rpm-base
>>> system you get installed gems as close as possible to what would you
>>> get on fedora 16. (you cannot install with bundle install only right 
>>> now)
>>>   * (optionally including the security patches, lets discuss this 
>>> later)
>>>
>>> = What was discussed, why I think it wont work
>>>
>>> == Branch for each platform
>>>
>>> * There would be little difference between the branches (only Gemfile
>>> or Gemfile.lock). Rest of the code will be same.
>>> * It would be too much work to maintain it.
>>>
>>> == Gemfile.lock per platform and environment
>>>
>>> * We would have to have 8 Gemfile.locks and maintain them all (4
>>> platforms multiplied by two environments)
>>> * Too much work unless its automated.
>>> * Automation would take time to setup.
>>>
>>> == Aeolus solution, bundler_ext [2]
>>>
>>> * Dependencies resolution is unaffected so 'bundle install' cannot be
>>> used to install development dependencies, it would ignore rpm-gems and
>>> install gems from gem-repo.
>>> * If bundler_ext is used for setup development environment, control
>>> over which versions are required is lost (requiring is not done by
>>> bundler)
>>> * It affects only how requirements are loaded [1], it's useful only
>>> for deployment.
>>> * (We could use it in deployment to get rid of bundler, but that is
>>> out of this discussion's scope see Goal.2)
>>>
>>> == bundle install --without default
>>>
>>> This installs only development gems, but:
>>>
>>> * If some of the development gems has a production gem as dependency
>>> (eg. ActiveSupport), the gem is reinstalled by bundler from a gem-repo
>>> over the older rpm-gem.
>>> * If only development gems are installed, bundler cannot be used to
>>> require gems. It will raise LoadError because from bundler's point of
>>> view production gems are not installed.
>>> * It would have to be combined with bundler_ext to find production
>>> rpm-gems, so you would loose control over which version are loaded 
>>> again
>>> * If you forget to add '--without default' when you run 'bundle
>>> install' unwanted versions are installed.
>>>
>>> == vendor/cache
>>>
>>> Move all gems to src/vendor/cache (directly into katello/master or as
>>> a git sub-module) and remove sources form Gemfile completely. 'bundle
>>> install' installs gems form vendor/cache automatically if there are no
>>> sources in Gemfile.
>>>
>>> * to have .gem files directly in katello/master is ugly
>>> * submodules are real pain
>> no need to use sub-modules
>>> * git is not that good with binary data
>>>
>> gems are mostly ruby, very not a lot of binary data.
>
> gems would be stored in vendor/cache as .gem files, one file per gem 
> which are actually tar.gz (binary). That is what I meant sorry about 
> the confusion.
>
>>> = My proposal
>>>
>>> == Bundler patch
>>>
>>> * it would be loaded in Gemfile, so every tool using bundler will 
>>> get it
>> I don't think a single Gemfile is going to solve the issue: we'll need
>> to relax version constraints in Gemfile, and add a bunch of ruby-version
>> specific gems...
>>>
>>> * it would ensure that if rpm-gem is present it's used even if there
>>> is newer version available in gem-repo
>>> * ensure that locally installed gems are not confused with rpm-gems
>> You didn't explain how you are planning to differentiate between the two
>> - system-wide bundler-/ruby-gem-installed gems and rpm-installed gems...
>
> I don't think this is a problem, see [1] how I did it.

This is not a good approach - you introduced platform-specific code into 
main Katello code base.

>>> * patch is optional, can be turned on/off by ENV variable or by options
>>> * (I am also considering to add optional check that all production
>>> gems are installed as rpms)
>>>
>>> Deployment is unaffected, all dependencies are satisfied by rpm-gems.
>>>
>>> When you run bundle install to get development environment on any
>>> rpm-based machine, all your rpms have priority and they are used. All
>>> missing development gems are installed from our gem-repo.
>> I think this is the crux of the issue. Once we have a custom gem
>> repository, the above step has very little practical value. A ruby gem
>> is a ruby gem no matter what place in the system it's installed in.
>
> IMO This has a big practical value. When you are fixing a bug you want 
> to be sure that you have same rpm-gems installed and required as in 
> production where the bug was discovered.

Do you understand how ruby loads gem dependencies? Can you explain the 
difference between rpm-installed gem, and bundler-installed gem (for 
system-wide installs)?

-d

>
>
>>>
>>> On non-rpm-based machine, 'bundle install' installs all gems
>>> production and development from our gem-repo.
>>>
>>> == Gem Repository
>>>
>>> Bundler patch will work only if all development and production gems
>>> are in the gem-repo. To get it fixed quickly 'bundle package' can be
>>> used to get all the gems. Missing gems can be then uploaded to
>>> katello-thirdparty (source for our gem-repo)
>>>
>>> = Summary
>>>
>>> I think that Bundler patch and updating gem-repo to contain all gems
>>> is the only one which can be done fast and will work on any machine in
>>> deployment and development. I tried to be thorough but if I missed
>>> something or if you have better ideas please let me know.
>> The issue with dependency version variability needs solving. The patch
>> itself has little practical value, bundle install from custom repository
>> is sufficient. The irony is that you are trying to solve the problem
>> Bundler has successfully solved awhile ago. I'm baffled as to why we
>> continue to insist on solving the problems the "katello" way.
>>
>> -d
>>
>>>
>>> Petr
>>>
>>> [1]
>>> https://github.com/aeolusproject/conductor/blob/master/src/config/application.rb#L21 
>>>
>>>
>>> [2] https://github.com/aeolus-incubator/bundler_ext
>>>
>>> _______________________________________________
>>> katello-devel mailing list
>>> katello-devel at redhat.com
>>> https://www.redhat.com/mailman/listinfo/katello-devel
>>
>>
>> _______________________________________________
>> katello-devel mailing list
>> katello-devel at redhat.com
>> https://www.redhat.com/mailman/listinfo/katello-devel
>
> [1] https://github.com/Katello/katello/pull/526