[katello-devel] Bundler vs rpm-gems - in more detail

Tue Aug 28 11:13:48 UTC 2012

On 28.08.12 11:08, Dmitri Dolguikh wrote:
> On 28/08/12 08:21 AM, Petr Chalupa wrote:
>>
>>
>> On 23.08.12 14:42, Dmitri Dolguikh wrote:
>>> On 23/08/12 12:56 PM, Petr Chalupa wrote:
>>>> My original email raised a lot of questions, I should wrote it with
>>>> more details. I apology for that. I'll try again and elaborate.
>>>>
>>>> = Goals
>>>>
>>>> 1) To be able to implement the solution in a week or so.
>>>> 2) Not to affect deployment, keep it same way as now.
>>>> 3) When you are installing development environment on any rpm-based
>>>> system rpm-gems are used even if they are older then gems in our
>>>> gem-repo. (this is borken right now)
>>>> 4) When you are installing development environment in any non-rpm-base
>>>> system you get installed gems as close as possible to what would you
>>>> get on fedora 16. (you cannot install with bundle install only right
>>>> now)
>>>>   * (optionally including the security patches, lets discuss this
>>>> later)
>>>>
>>>> = What was discussed, why I think it wont work
>>>>
>>>> == Branch for each platform
>>>>
>>>> * There would be little difference between the branches (only Gemfile
>>>> or Gemfile.lock). Rest of the code will be same.
>>>> * It would be too much work to maintain it.
>>>>
>>>> == Gemfile.lock per platform and environment
>>>>
>>>> * We would have to have 8 Gemfile.locks and maintain them all (4
>>>> platforms multiplied by two environments)
>>>> * Too much work unless its automated.
>>>> * Automation would take time to setup.
>>>>
>>>> == Aeolus solution, bundler_ext [2]
>>>>
>>>> * Dependencies resolution is unaffected so 'bundle install' cannot be
>>>> used to install development dependencies, it would ignore rpm-gems and
>>>> install gems from gem-repo.
>>>> * If bundler_ext is used for setup development environment, control
>>>> over which versions are required is lost (requiring is not done by
>>>> bundler)
>>>> * It affects only how requirements are loaded [1], it's useful only
>>>> for deployment.
>>>> * (We could use it in deployment to get rid of bundler, but that is
>>>> out of this discussion's scope see Goal.2)
>>>>
>>>> == bundle install --without default
>>>>
>>>> This installs only development gems, but:
>>>>
>>>> * If some of the development gems has a production gem as dependency
>>>> (eg. ActiveSupport), the gem is reinstalled by bundler from a gem-repo
>>>> over the older rpm-gem.
>>>> * If only development gems are installed, bundler cannot be used to
>>>> require gems. It will raise LoadError because from bundler's point of
>>>> view production gems are not installed.
>>>> * It would have to be combined with bundler_ext to find production
>>>> rpm-gems, so you would loose control over which version are loaded
>>>> again
>>>> * If you forget to add '--without default' when you run 'bundle
>>>> install' unwanted versions are installed.
>>>>
>>>> == vendor/cache
>>>>
>>>> Move all gems to src/vendor/cache (directly into katello/master or as
>>>> a git sub-module) and remove sources form Gemfile completely. 'bundle
>>>> install' installs gems form vendor/cache automatically if there are no
>>>> sources in Gemfile.
>>>>
>>>> * to have .gem files directly in katello/master is ugly
>>>> * submodules are real pain
>>> no need to use sub-modules
>>>> * git is not that good with binary data
>>>>
>>> gems are mostly ruby, very not a lot of binary data.
>>
>> gems would be stored in vendor/cache as .gem files, one file per gem
>> which are actually tar.gz (binary). That is what I meant sorry about
>> the confusion.
>>
>>>> = My proposal
>>>>
>>>> == Bundler patch
>>>>
>>>> * it would be loaded in Gemfile, so every tool using bundler will
>>>> get it
>>> I don't think a single Gemfile is going to solve the issue: we'll need
>>> to relax version constraints in Gemfile, and add a bunch of ruby-version
>>> specific gems...
>>>>
>>>> * it would ensure that if rpm-gem is present it's used even if there
>>>> is newer version available in gem-repo
>>>> * ensure that locally installed gems are not confused with rpm-gems
>>> You didn't explain how you are planning to differentiate between the two
>>> - system-wide bundler-/ruby-gem-installed gems and rpm-installed gems...
>>
>> I don't think this is a problem, see [1] how I did it.
>
> This is not a good approach - you introduced platform-specific code into
> main Katello code base.
>
>>>> * patch is optional, can be turned on/off by ENV variable or by options
>>>> * (I am also considering to add optional check that all production
>>>> gems are installed as rpms)
>>>>
>>>> Deployment is unaffected, all dependencies are satisfied by rpm-gems.
>>>>
>>>> When you run bundle install to get development environment on any
>>>> rpm-based machine, all your rpms have priority and they are used. All
>>>> missing development gems are installed from our gem-repo.
>>> I think this is the crux of the issue. Once we have a custom gem
>>> repository, the above step has very little practical value. A ruby gem
>>> is a ruby gem no matter what place in the system it's installed in.
>>
>> IMO This has a big practical value. When you are fixing a bug you want
>> to be sure that you have same rpm-gems installed and required as in
>> production where the bug was discovered.
>
> Do you understand how ruby loads gem dependencies? Can you explain the
> difference between rpm-installed gem, and bundler-installed gem (for
> system-wide installs)?
>

Yeah, I understand, We just don't understand each other :) How is this 
relevant to what I said before? I would like to give you an answer but I 
don't see the connection, please explain where are you heading with this.

> -d
>
>>
>>
>>>>
>>>> On non-rpm-based machine, 'bundle install' installs all gems
>>>> production and development from our gem-repo.
>>>>
>>>> == Gem Repository
>>>>
>>>> Bundler patch will work only if all development and production gems
>>>> are in the gem-repo. To get it fixed quickly 'bundle package' can be
>>>> used to get all the gems. Missing gems can be then uploaded to
>>>> katello-thirdparty (source for our gem-repo)
>>>>
>>>> = Summary
>>>>
>>>> I think that Bundler patch and updating gem-repo to contain all gems
>>>> is the only one which can be done fast and will work on any machine in
>>>> deployment and development. I tried to be thorough but if I missed
>>>> something or if you have better ideas please let me know.
>>> The issue with dependency version variability needs solving. The patch
>>> itself has little practical value, bundle install from custom repository
>>> is sufficient. The irony is that you are trying to solve the problem
>>> Bundler has successfully solved awhile ago. I'm baffled as to why we
>>> continue to insist on solving the problems the "katello" way.
>>>
>>> -d
>>>
>>>>
>>>> Petr
>>>>
>>>> [1]
>>>> https://github.com/aeolusproject/conductor/blob/master/src/config/application.rb#L21
>>>>
>>>>
>>>> [2] https://github.com/aeolus-incubator/bundler_ext
>>>>
>>>> _______________________________________________
>>>> katello-devel mailing list
>>>> katello-devel at redhat.com
>>>> https://www.redhat.com/mailman/listinfo/katello-devel
>>>
>>>
>>> _______________________________________________
>>> katello-devel mailing list
>>> katello-devel at redhat.com
>>> https://www.redhat.com/mailman/listinfo/katello-devel
>>
>> [1] https://github.com/Katello/katello/pull/526
>
>