[katello-devel] Bundler vs rpm-gems - in more detail

Tue Aug 28 13:09:16 UTC 2012

On 28.08.12 14:11, Dmitri Dolguikh wrote:
> On 28/08/12 01:04 PM, Petr Chalupa wrote:
>>
>>
>> On 28.08.12 13:53, Dmitri Dolguikh wrote:
>>> On 28/08/12 12:47 PM, Petr Chalupa wrote:
>>>>
>>>>
>>>> On Tue Aug 28 13:24:55 2012, Dmitri Dolguikh wrote:
>>>>> On 28/08/12 12:13 PM, Petr Chalupa wrote:
>>>>>>
>>>>>>
>>>>>> On 28.08.12 11:08, Dmitri Dolguikh wrote:
>>>>>>> On 28/08/12 08:21 AM, Petr Chalupa wrote:
>>>>>>>>
>>>>>>>>
>>>>>>>> On 23.08.12 14:42, Dmitri Dolguikh wrote:
>>>>>>>>> On 23/08/12 12:56 PM, Petr Chalupa wrote:
>>>>>>>>>> My original email raised a lot of questions, I should wrote it
>>>>>>>>>> with
>>>>>>>>>> more details. I apology for that. I'll try again and elaborate.
>>>>>>>>>>
>>>>>>>>>> = Goals
>>>>>>>>>>
>>>>>>>>>> 1) To be able to implement the solution in a week or so.
>>>>>>>>>> 2) Not to affect deployment, keep it same way as now.
>>>>>>>>>> 3) When you are installing development environment on any
>>>>>>>>>> rpm-based
>>>>>>>>>> system rpm-gems are used even if they are older then gems in our
>>>>>>>>>> gem-repo. (this is borken right now)
>>>>>>>>>> 4) When you are installing development environment in any
>>>>>>>>>> non-rpm-base
>>>>>>>>>> system you get installed gems as close as possible to what would
>>>>>>>>>> you
>>>>>>>>>> get on fedora 16. (you cannot install with bundle install only
>>>>>>>>>> right
>>>>>>>>>> now)
>>>>>>>>>>   * (optionally including the security patches, lets discuss this
>>>>>>>>>> later)
>>>>>>>>>>
>>>>>>>>>> = What was discussed, why I think it wont work
>>>>>>>>>>
>>>>>>>>>> == Branch for each platform
>>>>>>>>>>
>>>>>>>>>> * There would be little difference between the branches (only
>>>>>>>>>> Gemfile
>>>>>>>>>> or Gemfile.lock). Rest of the code will be same.
>>>>>>>>>> * It would be too much work to maintain it.
>>>>>>>>>>
>>>>>>>>>> == Gemfile.lock per platform and environment
>>>>>>>>>>
>>>>>>>>>> * We would have to have 8 Gemfile.locks and maintain them all (4
>>>>>>>>>> platforms multiplied by two environments)
>>>>>>>>>> * Too much work unless its automated.
>>>>>>>>>> * Automation would take time to setup.
>>>>>>>>>>
>>>>>>>>>> == Aeolus solution, bundler_ext [2]
>>>>>>>>>>
>>>>>>>>>> * Dependencies resolution is unaffected so 'bundle install'
>>>>>>>>>> cannot be
>>>>>>>>>> used to install development dependencies, it would ignore
>>>>>>>>>> rpm-gems and
>>>>>>>>>> install gems from gem-repo.
>>>>>>>>>> * If bundler_ext is used for setup development environment,
>>>>>>>>>> control
>>>>>>>>>> over which versions are required is lost (requiring is not
>>>>>>>>>> done by
>>>>>>>>>> bundler)
>>>>>>>>>> * It affects only how requirements are loaded [1], it's useful
>>>>>>>>>> only
>>>>>>>>>> for deployment.
>>>>>>>>>> * (We could use it in deployment to get rid of bundler, but
>>>>>>>>>> that is
>>>>>>>>>> out of this discussion's scope see Goal.2)
>>>>>>>>>>
>>>>>>>>>> == bundle install --without default
>>>>>>>>>>
>>>>>>>>>> This installs only development gems, but:
>>>>>>>>>>
>>>>>>>>>> * If some of the development gems has a production gem as
>>>>>>>>>> dependency
>>>>>>>>>> (eg. ActiveSupport), the gem is reinstalled by bundler from a
>>>>>>>>>> gem-repo
>>>>>>>>>> over the older rpm-gem.
>>>>>>>>>> * If only development gems are installed, bundler cannot be
>>>>>>>>>> used to
>>>>>>>>>> require gems. It will raise LoadError because from bundler's
>>>>>>>>>> point of
>>>>>>>>>> view production gems are not installed.
>>>>>>>>>> * It would have to be combined with bundler_ext to find
>>>>>>>>>> production
>>>>>>>>>> rpm-gems, so you would loose control over which version are
>>>>>>>>>> loaded
>>>>>>>>>> again
>>>>>>>>>> * If you forget to add '--without default' when you run 'bundle
>>>>>>>>>> install' unwanted versions are installed.
>>>>>>>>>>
>>>>>>>>>> == vendor/cache
>>>>>>>>>>
>>>>>>>>>> Move all gems to src/vendor/cache (directly into katello/master
>>>>>>>>>> or as
>>>>>>>>>> a git sub-module) and remove sources form Gemfile completely.
>>>>>>>>>> 'bundle
>>>>>>>>>> install' installs gems form vendor/cache automatically if there
>>>>>>>>>> are no
>>>>>>>>>> sources in Gemfile.
>>>>>>>>>>
>>>>>>>>>> * to have .gem files directly in katello/master is ugly
>>>>>>>>>> * submodules are real pain
>>>>>>>>> no need to use sub-modules
>>>>>>>>>> * git is not that good with binary data
>>>>>>>>>>
>>>>>>>>> gems are mostly ruby, very not a lot of binary data.
>>>>>>>>
>>>>>>>> gems would be stored in vendor/cache as .gem files, one file per
>>>>>>>> gem
>>>>>>>> which are actually tar.gz (binary). That is what I meant sorry
>>>>>>>> about
>>>>>>>> the confusion.
>>>>>>>>
>>>>>>>>>> = My proposal
>>>>>>>>>>
>>>>>>>>>> == Bundler patch
>>>>>>>>>>
>>>>>>>>>> * it would be loaded in Gemfile, so every tool using bundler will
>>>>>>>>>> get it
>>>>>>>>> I don't think a single Gemfile is going to solve the issue: we'll
>>>>>>>>> need
>>>>>>>>> to relax version constraints in Gemfile, and add a bunch of
>>>>>>>>> ruby-version
>>>>>>>>> specific gems...
>>>>>>>>>>
>>>>>>>>>> * it would ensure that if rpm-gem is present it's used even if
>>>>>>>>>> there
>>>>>>>>>> is newer version available in gem-repo
>>>>>>>>>> * ensure that locally installed gems are not confused with
>>>>>>>>>> rpm-gems
>>>>>>>>> You didn't explain how you are planning to differentiate between
>>>>>>>>> the two
>>>>>>>>> - system-wide bundler-/ruby-gem-installed gems and rpm-installed
>>>>>>>>> gems...
>>>>>>>>
>>>>>>>> I don't think this is a problem, see [1] how I did it.
>>>>>>>
>>>>>>> This is not a good approach - you introduced platform-specific code
>>>>>>> into
>>>>>>> main Katello code base.
>>>>>>>
>>>>>>>>>> * patch is optional, can be turned on/off by ENV variable or by
>>>>>>>>>> options
>>>>>>>>>> * (I am also considering to add optional check that all
>>>>>>>>>> production
>>>>>>>>>> gems are installed as rpms)
>>>>>>>>>>
>>>>>>>>>> Deployment is unaffected, all dependencies are satisfied by
>>>>>>>>>> rpm-gems.
>>>>>>>>>>
>>>>>>>>>> When you run bundle install to get development environment on any
>>>>>>>>>> rpm-based machine, all your rpms have priority and they are used.
>>>>>>>>>> All
>>>>>>>>>> missing development gems are installed from our gem-repo.
>>>>>>>>> I think this is the crux of the issue. Once we have a custom gem
>>>>>>>>> repository, the above step has very little practical value. A ruby
>>>>>>>>> gem
>>>>>>>>> is a ruby gem no matter what place in the system it's installed
>>>>>>>>> in.
>>>>>>>>
>>>>>>>> IMO This has a big practical value. When you are fixing a bug you
>>>>>>>> want
>>>>>>>> to be sure that you have same rpm-gems installed and required as in
>>>>>>>> production where the bug was discovered.
>>>>>>>
>>>>>>> Do you understand how ruby loads gem dependencies? Can you
>>>>>>> explain the
>>>>>>> difference between rpm-installed gem, and bundler-installed gem (for
>>>>>>> system-wide installs)?
>>>>>>>
>>>>>>
>>>>>> Yeah, I understand, We just don't understand each other :) How is
>>>>>> this relevant to what I said before? I would like to give you an
>>>>>> answer but I don't see the connection, please explain where are you
>>>>>> heading with this.
>>>>>
>>>>> Ultimately, one way, or another, we are solving the problem of loading
>>>>> gem dependencies. I'd like you to demonstrate the difference between
>>>>> loading of a regular ruby gem and rpm-installed ruby gem by ruby
>>>>> interpreter.
>>>>>
>>>>
>>>> The patch I introduced does not affect how gems are required. It
>>>> changes how bundler prioritizes gem matching against dependencies.
>>>> Rpm-gems are moved to the top in the queue of gem candidates for the
>>>> dependency matching. So rpm-gems are matched/picked before other gems.
>>>> Rpm-gems are matched/picked even if there is newer version in gem-repo.
>>>>
>>>> To answer you question, plain ruby process requires newest available
>>>> version of a given gem, regular or rpm-installed. There is no
>>>> difference for Ruby. This is unchanged by the patch.
>>>>
>>>> Ruby process using Bundler (with or without the patch) to require gems
>>>> will require gems in versions as they are specified in Gemfile.lock,
>>>> regular or rpm-installed. This is also unchanged by the patch.
>>>>
>>>> The ruby requiring algorithm with or without bundler is unchanged by
>>>> the patch.
>>>>
>>>> What gems are in Gemfile.lock is resolved when `bundle install` is
>>>> run. What gems are resolved and picked to Gemfile.lock is altered by
>>>> the patch.
>>>>
>>>> Does this help to answer your question?
>>> It does, thank you. You have shown there's no difference between loading
>>> gems from either of the sources. What benefits does your patch provide
>>> over bundler using have platform-specific repository[ies]?
>>>
>>> -d
>>
>> You are able to install development dependencies on top of already
>> installed rpm-gems (production dependencies) without overshadowing
>> them with newer gems from gem-repo. Needed for RHEL and CentOS. On
>> fedora 16 the rpm-gems have same versions as gems in our gem-repo. On
>> non-rpm machine you get same versions as on fedora 16 because I also
>> updated gem-repo with missing gems (this was not possible before).
>>
>
> I thought we were going to have platform-specific gem repositories which
> would contain platform-appropriate versions of gems? If we were to have
> just one repository, how would you solve the problem of development
> dependency requiring newer production dependency not available on the
> target platform?
>

This was suggested by Lukas, I didn't intended to implement it now. I 
wanted quick-fix in 1 or 2 days. (This patch and adding missing gems to 
gem-repo were quick solution).

The platform specific gems are kept (on fedora or rhel) because rpm-gems 
are preferred before regular gems.

Seems to me that the problem you are describing is exotic and if we 
encounter something like that we should avoid it by specifying 
dependency on older version of the development gem.

>> Petr
>>
>>>>
>>>> Petr
>>>>
>>>>>
>>>>>>
>>>>>>> -d
>>>>>>>
>>>>>>>>
>>>>>>>>
>>>>>>>>>>
>>>>>>>>>> On non-rpm-based machine, 'bundle install' installs all gems
>>>>>>>>>> production and development from our gem-repo.
>>>>>>>>>>
>>>>>>>>>> == Gem Repository
>>>>>>>>>>
>>>>>>>>>> Bundler patch will work only if all development and production
>>>>>>>>>> gems
>>>>>>>>>> are in the gem-repo. To get it fixed quickly 'bundle package'
>>>>>>>>>> can be
>>>>>>>>>> used to get all the gems. Missing gems can be then uploaded to
>>>>>>>>>> katello-thirdparty (source for our gem-repo)
>>>>>>>>>>
>>>>>>>>>> = Summary
>>>>>>>>>>
>>>>>>>>>> I think that Bundler patch and updating gem-repo to contain all
>>>>>>>>>> gems
>>>>>>>>>> is the only one which can be done fast and will work on any
>>>>>>>>>> machine in
>>>>>>>>>> deployment and development. I tried to be thorough but if I
>>>>>>>>>> missed
>>>>>>>>>> something or if you have better ideas please let me know.
>>>>>>>>> The issue with dependency version variability needs solving. The
>>>>>>>>> patch
>>>>>>>>> itself has little practical value, bundle install from custom
>>>>>>>>> repository
>>>>>>>>> is sufficient. The irony is that you are trying to solve the
>>>>>>>>> problem
>>>>>>>>> Bundler has successfully solved awhile ago. I'm baffled as to
>>>>>>>>> why we
>>>>>>>>> continue to insist on solving the problems the "katello" way.
>>>>>>>>>
>>>>>>>>> -d
>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>> Petr
>>>>>>>>>>
>>>>>>>>>> [1]
>>>>>>>>>> https://github.com/aeolusproject/conductor/blob/master/src/config/application.rb#L21
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>> [2] https://github.com/aeolus-incubator/bundler_ext
>>>>>>>>>>
>>>>>>>>>> _______________________________________________
>>>>>>>>>> katello-devel mailing list
>>>>>>>>>> katello-devel at redhat.com
>>>>>>>>>> https://www.redhat.com/mailman/listinfo/katello-devel
>>>>>>>>>
>>>>>>>>>
>>>>>>>>> _______________________________________________
>>>>>>>>> katello-devel mailing list
>>>>>>>>> katello-devel at redhat.com
>>>>>>>>> https://www.redhat.com/mailman/listinfo/katello-devel
>>>>>>>>
>>>>>>>> [1] https://github.com/Katello/katello/pull/526
>>>>>>>
>>>>>>>
>>>>>
>>>>>
>>>
>>>
>
>