[katello-devel] Bundler vs rpm-gems - in more detail

Tue Aug 28 12:04:27 UTC 2012

On 28.08.12 13:53, Dmitri Dolguikh wrote:
> On 28/08/12 12:47 PM, Petr Chalupa wrote:
>>
>>
>> On Tue Aug 28 13:24:55 2012, Dmitri Dolguikh wrote:
>>> On 28/08/12 12:13 PM, Petr Chalupa wrote:
>>>>
>>>>
>>>> On 28.08.12 11:08, Dmitri Dolguikh wrote:
>>>>> On 28/08/12 08:21 AM, Petr Chalupa wrote:
>>>>>>
>>>>>>
>>>>>> On 23.08.12 14:42, Dmitri Dolguikh wrote:
>>>>>>> On 23/08/12 12:56 PM, Petr Chalupa wrote:
>>>>>>>> My original email raised a lot of questions, I should wrote it with
>>>>>>>> more details. I apology for that. I'll try again and elaborate.
>>>>>>>>
>>>>>>>> = Goals
>>>>>>>>
>>>>>>>> 1) To be able to implement the solution in a week or so.
>>>>>>>> 2) Not to affect deployment, keep it same way as now.
>>>>>>>> 3) When you are installing development environment on any rpm-based
>>>>>>>> system rpm-gems are used even if they are older then gems in our
>>>>>>>> gem-repo. (this is borken right now)
>>>>>>>> 4) When you are installing development environment in any
>>>>>>>> non-rpm-base
>>>>>>>> system you get installed gems as close as possible to what would
>>>>>>>> you
>>>>>>>> get on fedora 16. (you cannot install with bundle install only
>>>>>>>> right
>>>>>>>> now)
>>>>>>>>   * (optionally including the security patches, lets discuss this
>>>>>>>> later)
>>>>>>>>
>>>>>>>> = What was discussed, why I think it wont work
>>>>>>>>
>>>>>>>> == Branch for each platform
>>>>>>>>
>>>>>>>> * There would be little difference between the branches (only
>>>>>>>> Gemfile
>>>>>>>> or Gemfile.lock). Rest of the code will be same.
>>>>>>>> * It would be too much work to maintain it.
>>>>>>>>
>>>>>>>> == Gemfile.lock per platform and environment
>>>>>>>>
>>>>>>>> * We would have to have 8 Gemfile.locks and maintain them all (4
>>>>>>>> platforms multiplied by two environments)
>>>>>>>> * Too much work unless its automated.
>>>>>>>> * Automation would take time to setup.
>>>>>>>>
>>>>>>>> == Aeolus solution, bundler_ext [2]
>>>>>>>>
>>>>>>>> * Dependencies resolution is unaffected so 'bundle install'
>>>>>>>> cannot be
>>>>>>>> used to install development dependencies, it would ignore
>>>>>>>> rpm-gems and
>>>>>>>> install gems from gem-repo.
>>>>>>>> * If bundler_ext is used for setup development environment, control
>>>>>>>> over which versions are required is lost (requiring is not done by
>>>>>>>> bundler)
>>>>>>>> * It affects only how requirements are loaded [1], it's useful only
>>>>>>>> for deployment.
>>>>>>>> * (We could use it in deployment to get rid of bundler, but that is
>>>>>>>> out of this discussion's scope see Goal.2)
>>>>>>>>
>>>>>>>> == bundle install --without default
>>>>>>>>
>>>>>>>> This installs only development gems, but:
>>>>>>>>
>>>>>>>> * If some of the development gems has a production gem as
>>>>>>>> dependency
>>>>>>>> (eg. ActiveSupport), the gem is reinstalled by bundler from a
>>>>>>>> gem-repo
>>>>>>>> over the older rpm-gem.
>>>>>>>> * If only development gems are installed, bundler cannot be used to
>>>>>>>> require gems. It will raise LoadError because from bundler's
>>>>>>>> point of
>>>>>>>> view production gems are not installed.
>>>>>>>> * It would have to be combined with bundler_ext to find production
>>>>>>>> rpm-gems, so you would loose control over which version are loaded
>>>>>>>> again
>>>>>>>> * If you forget to add '--without default' when you run 'bundle
>>>>>>>> install' unwanted versions are installed.
>>>>>>>>
>>>>>>>> == vendor/cache
>>>>>>>>
>>>>>>>> Move all gems to src/vendor/cache (directly into katello/master
>>>>>>>> or as
>>>>>>>> a git sub-module) and remove sources form Gemfile completely.
>>>>>>>> 'bundle
>>>>>>>> install' installs gems form vendor/cache automatically if there
>>>>>>>> are no
>>>>>>>> sources in Gemfile.
>>>>>>>>
>>>>>>>> * to have .gem files directly in katello/master is ugly
>>>>>>>> * submodules are real pain
>>>>>>> no need to use sub-modules
>>>>>>>> * git is not that good with binary data
>>>>>>>>
>>>>>>> gems are mostly ruby, very not a lot of binary data.
>>>>>>
>>>>>> gems would be stored in vendor/cache as .gem files, one file per gem
>>>>>> which are actually tar.gz (binary). That is what I meant sorry about
>>>>>> the confusion.
>>>>>>
>>>>>>>> = My proposal
>>>>>>>>
>>>>>>>> == Bundler patch
>>>>>>>>
>>>>>>>> * it would be loaded in Gemfile, so every tool using bundler will
>>>>>>>> get it
>>>>>>> I don't think a single Gemfile is going to solve the issue: we'll
>>>>>>> need
>>>>>>> to relax version constraints in Gemfile, and add a bunch of
>>>>>>> ruby-version
>>>>>>> specific gems...
>>>>>>>>
>>>>>>>> * it would ensure that if rpm-gem is present it's used even if
>>>>>>>> there
>>>>>>>> is newer version available in gem-repo
>>>>>>>> * ensure that locally installed gems are not confused with rpm-gems
>>>>>>> You didn't explain how you are planning to differentiate between
>>>>>>> the two
>>>>>>> - system-wide bundler-/ruby-gem-installed gems and rpm-installed
>>>>>>> gems...
>>>>>>
>>>>>> I don't think this is a problem, see [1] how I did it.
>>>>>
>>>>> This is not a good approach - you introduced platform-specific code
>>>>> into
>>>>> main Katello code base.
>>>>>
>>>>>>>> * patch is optional, can be turned on/off by ENV variable or by
>>>>>>>> options
>>>>>>>> * (I am also considering to add optional check that all production
>>>>>>>> gems are installed as rpms)
>>>>>>>>
>>>>>>>> Deployment is unaffected, all dependencies are satisfied by
>>>>>>>> rpm-gems.
>>>>>>>>
>>>>>>>> When you run bundle install to get development environment on any
>>>>>>>> rpm-based machine, all your rpms have priority and they are used.
>>>>>>>> All
>>>>>>>> missing development gems are installed from our gem-repo.
>>>>>>> I think this is the crux of the issue. Once we have a custom gem
>>>>>>> repository, the above step has very little practical value. A ruby
>>>>>>> gem
>>>>>>> is a ruby gem no matter what place in the system it's installed in.
>>>>>>
>>>>>> IMO This has a big practical value. When you are fixing a bug you
>>>>>> want
>>>>>> to be sure that you have same rpm-gems installed and required as in
>>>>>> production where the bug was discovered.
>>>>>
>>>>> Do you understand how ruby loads gem dependencies? Can you explain the
>>>>> difference between rpm-installed gem, and bundler-installed gem (for
>>>>> system-wide installs)?
>>>>>
>>>>
>>>> Yeah, I understand, We just don't understand each other :) How is
>>>> this relevant to what I said before? I would like to give you an
>>>> answer but I don't see the connection, please explain where are you
>>>> heading with this.
>>>
>>> Ultimately, one way, or another, we are solving the problem of loading
>>> gem dependencies. I'd like you to demonstrate the difference between
>>> loading of a regular ruby gem and rpm-installed ruby gem by ruby
>>> interpreter.
>>>
>>
>> The patch I introduced does not affect how gems are required. It
>> changes how bundler prioritizes gem matching against dependencies.
>> Rpm-gems are moved to the top in the queue of gem candidates for the
>> dependency matching. So rpm-gems are matched/picked before other gems.
>> Rpm-gems are matched/picked even if there is newer version in gem-repo.
>>
>> To answer you question, plain ruby process requires newest available
>> version of a given gem, regular or rpm-installed. There is no
>> difference for Ruby. This is unchanged by the patch.
>>
>> Ruby process using Bundler (with or without the patch) to require gems
>> will require gems in versions as they are specified in Gemfile.lock,
>> regular or rpm-installed. This is also unchanged by the patch.
>>
>> The ruby requiring algorithm with or without bundler is unchanged by
>> the patch.
>>
>> What gems are in Gemfile.lock is resolved when `bundle install` is
>> run. What gems are resolved and picked to Gemfile.lock is altered by
>> the patch.
>>
>> Does this help to answer your question?
> It does, thank you. You have shown there's no difference between loading
> gems from either of the sources. What benefits does your patch provide
> over bundler using have platform-specific repository[ies]?
>
> -d

You are able to install development dependencies on top of already 
installed rpm-gems (production dependencies) without overshadowing them 
with newer gems from gem-repo. Needed for RHEL and CentOS. On fedora 16 
the rpm-gems have same versions as gems in our gem-repo. On non-rpm 
machine you get same versions as on fedora 16 because I also updated 
gem-repo with missing gems (this was not possible before).

Petr

>>
>> Petr
>>
>>>
>>>>
>>>>> -d
>>>>>
>>>>>>
>>>>>>
>>>>>>>>
>>>>>>>> On non-rpm-based machine, 'bundle install' installs all gems
>>>>>>>> production and development from our gem-repo.
>>>>>>>>
>>>>>>>> == Gem Repository
>>>>>>>>
>>>>>>>> Bundler patch will work only if all development and production gems
>>>>>>>> are in the gem-repo. To get it fixed quickly 'bundle package'
>>>>>>>> can be
>>>>>>>> used to get all the gems. Missing gems can be then uploaded to
>>>>>>>> katello-thirdparty (source for our gem-repo)
>>>>>>>>
>>>>>>>> = Summary
>>>>>>>>
>>>>>>>> I think that Bundler patch and updating gem-repo to contain all
>>>>>>>> gems
>>>>>>>> is the only one which can be done fast and will work on any
>>>>>>>> machine in
>>>>>>>> deployment and development. I tried to be thorough but if I missed
>>>>>>>> something or if you have better ideas please let me know.
>>>>>>> The issue with dependency version variability needs solving. The
>>>>>>> patch
>>>>>>> itself has little practical value, bundle install from custom
>>>>>>> repository
>>>>>>> is sufficient. The irony is that you are trying to solve the problem
>>>>>>> Bundler has successfully solved awhile ago. I'm baffled as to why we
>>>>>>> continue to insist on solving the problems the "katello" way.
>>>>>>>
>>>>>>> -d
>>>>>>>
>>>>>>>>
>>>>>>>> Petr
>>>>>>>>
>>>>>>>> [1]
>>>>>>>> https://github.com/aeolusproject/conductor/blob/master/src/config/application.rb#L21
>>>>>>>>
>>>>>>>>
>>>>>>>>
>>>>>>>>
>>>>>>>> [2] https://github.com/aeolus-incubator/bundler_ext
>>>>>>>>
>>>>>>>> _______________________________________________
>>>>>>>> katello-devel mailing list
>>>>>>>> katello-devel at redhat.com
>>>>>>>> https://www.redhat.com/mailman/listinfo/katello-devel
>>>>>>>
>>>>>>>
>>>>>>> _______________________________________________
>>>>>>> katello-devel mailing list
>>>>>>> katello-devel at redhat.com
>>>>>>> https://www.redhat.com/mailman/listinfo/katello-devel
>>>>>>
>>>>>> [1] https://github.com/Katello/katello/pull/526
>>>>>
>>>>>
>>>
>>>
>
>