[katello-devel] Bundler vs rpm-gems - in more detail

Tue Aug 28 14:17:46 UTC 2012

On 28/08/12 02:09 PM, Petr Chalupa wrote:
>
>
> On 28.08.12 14:11, Dmitri Dolguikh wrote:
>> On 28/08/12 01:04 PM, Petr Chalupa wrote:
>>>
>>>
>>> On 28.08.12 13:53, Dmitri Dolguikh wrote:
>>>> On 28/08/12 12:47 PM, Petr Chalupa wrote:
>>>>>
>>>>>
>>>>> On Tue Aug 28 13:24:55 2012, Dmitri Dolguikh wrote:
>>>>>> On 28/08/12 12:13 PM, Petr Chalupa wrote:
>>>>>>>
>>>>>>>
>>>>>>> On 28.08.12 11:08, Dmitri Dolguikh wrote:
>>>>>>>> On 28/08/12 08:21 AM, Petr Chalupa wrote:
>>>>>>>>>
>>>>>>>>>
>>>>>>>>> On 23.08.12 14:42, Dmitri Dolguikh wrote:
>>>>>>>>>> On 23/08/12 12:56 PM, Petr Chalupa wrote:
>>>>>>>>>>> My original email raised a lot of questions, I should wrote it
>>>>>>>>>>> with
>>>>>>>>>>> more details. I apology for that. I'll try again and elaborate.
>>>>>>>>>>>
>>>>>>>>>>> = Goals
>>>>>>>>>>>
>>>>>>>>>>> 1) To be able to implement the solution in a week or so.
>>>>>>>>>>> 2) Not to affect deployment, keep it same way as now.
>>>>>>>>>>> 3) When you are installing development environment on any
>>>>>>>>>>> rpm-based
>>>>>>>>>>> system rpm-gems are used even if they are older then gems in 
>>>>>>>>>>> our
>>>>>>>>>>> gem-repo. (this is borken right now)
>>>>>>>>>>> 4) When you are installing development environment in any
>>>>>>>>>>> non-rpm-base
>>>>>>>>>>> system you get installed gems as close as possible to what 
>>>>>>>>>>> would
>>>>>>>>>>> you
>>>>>>>>>>> get on fedora 16. (you cannot install with bundle install only
>>>>>>>>>>> right
>>>>>>>>>>> now)
>>>>>>>>>>>   * (optionally including the security patches, lets discuss 
>>>>>>>>>>> this
>>>>>>>>>>> later)
>>>>>>>>>>>
>>>>>>>>>>> = What was discussed, why I think it wont work
>>>>>>>>>>>
>>>>>>>>>>> == Branch for each platform
>>>>>>>>>>>
>>>>>>>>>>> * There would be little difference between the branches (only
>>>>>>>>>>> Gemfile
>>>>>>>>>>> or Gemfile.lock). Rest of the code will be same.
>>>>>>>>>>> * It would be too much work to maintain it.
>>>>>>>>>>>
>>>>>>>>>>> == Gemfile.lock per platform and environment
>>>>>>>>>>>
>>>>>>>>>>> * We would have to have 8 Gemfile.locks and maintain them 
>>>>>>>>>>> all (4
>>>>>>>>>>> platforms multiplied by two environments)
>>>>>>>>>>> * Too much work unless its automated.
>>>>>>>>>>> * Automation would take time to setup.
>>>>>>>>>>>
>>>>>>>>>>> == Aeolus solution, bundler_ext [2]
>>>>>>>>>>>
>>>>>>>>>>> * Dependencies resolution is unaffected so 'bundle install'
>>>>>>>>>>> cannot be
>>>>>>>>>>> used to install development dependencies, it would ignore
>>>>>>>>>>> rpm-gems and
>>>>>>>>>>> install gems from gem-repo.
>>>>>>>>>>> * If bundler_ext is used for setup development environment,
>>>>>>>>>>> control
>>>>>>>>>>> over which versions are required is lost (requiring is not
>>>>>>>>>>> done by
>>>>>>>>>>> bundler)
>>>>>>>>>>> * It affects only how requirements are loaded [1], it's useful
>>>>>>>>>>> only
>>>>>>>>>>> for deployment.
>>>>>>>>>>> * (We could use it in deployment to get rid of bundler, but
>>>>>>>>>>> that is
>>>>>>>>>>> out of this discussion's scope see Goal.2)
>>>>>>>>>>>
>>>>>>>>>>> == bundle install --without default
>>>>>>>>>>>
>>>>>>>>>>> This installs only development gems, but:
>>>>>>>>>>>
>>>>>>>>>>> * If some of the development gems has a production gem as
>>>>>>>>>>> dependency
>>>>>>>>>>> (eg. ActiveSupport), the gem is reinstalled by bundler from a
>>>>>>>>>>> gem-repo
>>>>>>>>>>> over the older rpm-gem.
>>>>>>>>>>> * If only development gems are installed, bundler cannot be
>>>>>>>>>>> used to
>>>>>>>>>>> require gems. It will raise LoadError because from bundler's
>>>>>>>>>>> point of
>>>>>>>>>>> view production gems are not installed.
>>>>>>>>>>> * It would have to be combined with bundler_ext to find
>>>>>>>>>>> production
>>>>>>>>>>> rpm-gems, so you would loose control over which version are
>>>>>>>>>>> loaded
>>>>>>>>>>> again
>>>>>>>>>>> * If you forget to add '--without default' when you run 'bundle
>>>>>>>>>>> install' unwanted versions are installed.
>>>>>>>>>>>
>>>>>>>>>>> == vendor/cache
>>>>>>>>>>>
>>>>>>>>>>> Move all gems to src/vendor/cache (directly into katello/master
>>>>>>>>>>> or as
>>>>>>>>>>> a git sub-module) and remove sources form Gemfile completely.
>>>>>>>>>>> 'bundle
>>>>>>>>>>> install' installs gems form vendor/cache automatically if there
>>>>>>>>>>> are no
>>>>>>>>>>> sources in Gemfile.
>>>>>>>>>>>
>>>>>>>>>>> * to have .gem files directly in katello/master is ugly
>>>>>>>>>>> * submodules are real pain
>>>>>>>>>> no need to use sub-modules
>>>>>>>>>>> * git is not that good with binary data
>>>>>>>>>>>
>>>>>>>>>> gems are mostly ruby, very not a lot of binary data.
>>>>>>>>>
>>>>>>>>> gems would be stored in vendor/cache as .gem files, one file per
>>>>>>>>> gem
>>>>>>>>> which are actually tar.gz (binary). That is what I meant sorry
>>>>>>>>> about
>>>>>>>>> the confusion.
>>>>>>>>>
>>>>>>>>>>> = My proposal
>>>>>>>>>>>
>>>>>>>>>>> == Bundler patch
>>>>>>>>>>>
>>>>>>>>>>> * it would be loaded in Gemfile, so every tool using bundler 
>>>>>>>>>>> will
>>>>>>>>>>> get it
>>>>>>>>>> I don't think a single Gemfile is going to solve the issue: 
>>>>>>>>>> we'll
>>>>>>>>>> need
>>>>>>>>>> to relax version constraints in Gemfile, and add a bunch of
>>>>>>>>>> ruby-version
>>>>>>>>>> specific gems...
>>>>>>>>>>>
>>>>>>>>>>> * it would ensure that if rpm-gem is present it's used even if
>>>>>>>>>>> there
>>>>>>>>>>> is newer version available in gem-repo
>>>>>>>>>>> * ensure that locally installed gems are not confused with
>>>>>>>>>>> rpm-gems
>>>>>>>>>> You didn't explain how you are planning to differentiate between
>>>>>>>>>> the two
>>>>>>>>>> - system-wide bundler-/ruby-gem-installed gems and rpm-installed
>>>>>>>>>> gems...
>>>>>>>>>
>>>>>>>>> I don't think this is a problem, see [1] how I did it.
>>>>>>>>
>>>>>>>> This is not a good approach - you introduced platform-specific 
>>>>>>>> code
>>>>>>>> into
>>>>>>>> main Katello code base.
>>>>>>>>
>>>>>>>>>>> * patch is optional, can be turned on/off by ENV variable or by
>>>>>>>>>>> options
>>>>>>>>>>> * (I am also considering to add optional check that all
>>>>>>>>>>> production
>>>>>>>>>>> gems are installed as rpms)
>>>>>>>>>>>
>>>>>>>>>>> Deployment is unaffected, all dependencies are satisfied by
>>>>>>>>>>> rpm-gems.
>>>>>>>>>>>
>>>>>>>>>>> When you run bundle install to get development environment 
>>>>>>>>>>> on any
>>>>>>>>>>> rpm-based machine, all your rpms have priority and they are 
>>>>>>>>>>> used.
>>>>>>>>>>> All
>>>>>>>>>>> missing development gems are installed from our gem-repo.
>>>>>>>>>> I think this is the crux of the issue. Once we have a custom gem
>>>>>>>>>> repository, the above step has very little practical value. A 
>>>>>>>>>> ruby
>>>>>>>>>> gem
>>>>>>>>>> is a ruby gem no matter what place in the system it's installed
>>>>>>>>>> in.
>>>>>>>>>
>>>>>>>>> IMO This has a big practical value. When you are fixing a bug you
>>>>>>>>> want
>>>>>>>>> to be sure that you have same rpm-gems installed and required 
>>>>>>>>> as in
>>>>>>>>> production where the bug was discovered.
>>>>>>>>
>>>>>>>> Do you understand how ruby loads gem dependencies? Can you
>>>>>>>> explain the
>>>>>>>> difference between rpm-installed gem, and bundler-installed gem 
>>>>>>>> (for
>>>>>>>> system-wide installs)?
>>>>>>>>
>>>>>>>
>>>>>>> Yeah, I understand, We just don't understand each other :) How is
>>>>>>> this relevant to what I said before? I would like to give you an
>>>>>>> answer but I don't see the connection, please explain where are you
>>>>>>> heading with this.
>>>>>>
>>>>>> Ultimately, one way, or another, we are solving the problem of 
>>>>>> loading
>>>>>> gem dependencies. I'd like you to demonstrate the difference between
>>>>>> loading of a regular ruby gem and rpm-installed ruby gem by ruby
>>>>>> interpreter.
>>>>>>
>>>>>
>>>>> The patch I introduced does not affect how gems are required. It
>>>>> changes how bundler prioritizes gem matching against dependencies.
>>>>> Rpm-gems are moved to the top in the queue of gem candidates for the
>>>>> dependency matching. So rpm-gems are matched/picked before other 
>>>>> gems.
>>>>> Rpm-gems are matched/picked even if there is newer version in 
>>>>> gem-repo.
>>>>>
>>>>> To answer you question, plain ruby process requires newest available
>>>>> version of a given gem, regular or rpm-installed. There is no
>>>>> difference for Ruby. This is unchanged by the patch.
>>>>>
>>>>> Ruby process using Bundler (with or without the patch) to require 
>>>>> gems
>>>>> will require gems in versions as they are specified in Gemfile.lock,
>>>>> regular or rpm-installed. This is also unchanged by the patch.
>>>>>
>>>>> The ruby requiring algorithm with or without bundler is unchanged by
>>>>> the patch.
>>>>>
>>>>> What gems are in Gemfile.lock is resolved when `bundle install` is
>>>>> run. What gems are resolved and picked to Gemfile.lock is altered by
>>>>> the patch.
>>>>>
>>>>> Does this help to answer your question?
>>>> It does, thank you. You have shown there's no difference between 
>>>> loading
>>>> gems from either of the sources. What benefits does your patch provide
>>>> over bundler using have platform-specific repository[ies]?
>>>>
>>>> -d
>>>
>>> You are able to install development dependencies on top of already
>>> installed rpm-gems (production dependencies) without overshadowing
>>> them with newer gems from gem-repo. Needed for RHEL and CentOS. On
>>> fedora 16 the rpm-gems have same versions as gems in our gem-repo. On
>>> non-rpm machine you get same versions as on fedora 16 because I also
>>> updated gem-repo with missing gems (this was not possible before).
>>>
>>
>> I thought we were going to have platform-specific gem repositories which
>> would contain platform-appropriate versions of gems? If we were to have
>> just one repository, how would you solve the problem of development
>> dependency requiring newer production dependency not available on the
>> target platform?
>>
>
> This was suggested by Lukas, I didn't intended to implement it now. I 
> wanted quick-fix in 1 or 2 days. (This patch and adding missing gems 
> to gem-repo were quick solution).

I'd like to have an idea about a long-term solution as well.

> The platform specific gems are kept (on fedora or rhel) because 
> rpm-gems are preferred before regular gems.
>
> Seems to me that the problem you are describing is exotic and if we 
> encounter something like that we should avoid it by specifying 
> dependency on older version of the development gem.

I think I see now. The approach you are suggesting *is* the simplest, 
provided you discount the effort of installing a given target platform 
for development, and the changes between platforms are minimal.

-d

>
>>> Petr
>>>
>>>>>
>>>>> Petr
>>>>>
>>>>>>
>>>>>>>
>>>>>>>> -d
>>>>>>>>
>>>>>>>>>
>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>> On non-rpm-based machine, 'bundle install' installs all gems
>>>>>>>>>>> production and development from our gem-repo.
>>>>>>>>>>>
>>>>>>>>>>> == Gem Repository
>>>>>>>>>>>
>>>>>>>>>>> Bundler patch will work only if all development and production
>>>>>>>>>>> gems
>>>>>>>>>>> are in the gem-repo. To get it fixed quickly 'bundle package'
>>>>>>>>>>> can be
>>>>>>>>>>> used to get all the gems. Missing gems can be then uploaded to
>>>>>>>>>>> katello-thirdparty (source for our gem-repo)
>>>>>>>>>>>
>>>>>>>>>>> = Summary
>>>>>>>>>>>
>>>>>>>>>>> I think that Bundler patch and updating gem-repo to contain all
>>>>>>>>>>> gems
>>>>>>>>>>> is the only one which can be done fast and will work on any
>>>>>>>>>>> machine in
>>>>>>>>>>> deployment and development. I tried to be thorough but if I
>>>>>>>>>>> missed
>>>>>>>>>>> something or if you have better ideas please let me know.
>>>>>>>>>> The issue with dependency version variability needs solving. The
>>>>>>>>>> patch
>>>>>>>>>> itself has little practical value, bundle install from custom
>>>>>>>>>> repository
>>>>>>>>>> is sufficient. The irony is that you are trying to solve the
>>>>>>>>>> problem
>>>>>>>>>> Bundler has successfully solved awhile ago. I'm baffled as to
>>>>>>>>>> why we
>>>>>>>>>> continue to insist on solving the problems the "katello" way.
>>>>>>>>>>
>>>>>>>>>> -d
>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>> Petr
>>>>>>>>>>>
>>>>>>>>>>> [1]
>>>>>>>>>>> https://github.com/aeolusproject/conductor/blob/master/src/config/application.rb#L21 
>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>> [2] https://github.com/aeolus-incubator/bundler_ext
>>>>>>>>>>>
>>>>>>>>>>> _______________________________________________
>>>>>>>>>>> katello-devel mailing list
>>>>>>>>>>> katello-devel at redhat.com
>>>>>>>>>>> https://www.redhat.com/mailman/listinfo/katello-devel
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>> _______________________________________________
>>>>>>>>>> katello-devel mailing list
>>>>>>>>>> katello-devel at redhat.com
>>>>>>>>>> https://www.redhat.com/mailman/listinfo/katello-devel
>>>>>>>>>
>>>>>>>>> [1] https://github.com/Katello/katello/pull/526
>>>>>>>>
>>>>>>>>
>>>>>>
>>>>>>
>>>>
>>>>
>>
>>