Ruby packaging next


#1

Taking ruby packaging to the next level

Table of Content

  1. TL;DR
  2. Where we started
  3. The basics
  4. One step at a time
  5. Rocks on the road
  6. “Job done right?” “Well almost.”
  7. Whats left?

TL;DR

  • we are going back to versioned ruby packages with a new naming scheme.
  • one spec file to rule them all. (one spec file to build a rubygem for
    all interpreters)
  • macro based buildrequires. %{rubygem rails:4.1 >= 4.1.4}
  • gem2rpm.yml as configuration file for gem2rpm. no more manual editing
    of spec files.
  • with the config spec files can always be regenerated without losing
    things.

Where we started

A long time ago we started with the support for building for multiple
ruby versions, actually MRI versions, at the same time. The ruby base
package was in good shape in that regard. But we had one open issue -
building rubygems for multiple ruby versions. This issue was hanging for
awhile so we went and reverted to a single ruby version packaged for
openSUSE again.

While this might work for openSUSE, it can become really challenging for
SLES with its long life cycle. Even in the openSUSE world we have
Evergreen for 11.4 still alive. That was released in March 2011 and
still no sign that Evergreen support for it is ending. We really want
multiple ruby versions here.

With that in mind we were wondering how much work would be left to do,
so we can actually fully support multiple ruby versions or even multiple
ruby interpreters.

The steps we need to take

When asking around for “If we support multiple versions again what would
you expect in regard to rubygems packaging?”, we got a few important
points.

  • only 1 spec file for all. python currently uses 1 spec file for each
    version
  • avoid re-packaging in d:l:r:e [1]
  • system ruby should always be /usr/bin/ruby.

The basics

The first step was to restore the versioned ruby packaging and add at
least one extra version. As a start we used a snapshot package for MRI
2.2.

All the common bits needed for all ruby interpreters moved into
ruby-common. The unversioned ruby package became a wrapper to pull in
the system ruby. As mentioned above /usr/bin/ruby should always point
to the system ruby. So in the new schema it is not a symlink
handled by update-alternatives, but a hardlink.

The same goes for %{_libdir}/libruby.so. It is actually a pity that we
still need this file, as any good program linking ruby should just query
RbConfig for the commands to link the libruby for the currently running
interpreter. Sadly some tools ask for the CFLAGS but then hardcode
-lruby. Ugh.

Our naming scheme is

mainpackage = "#{interpreter}#{major}.#{minor}"

jruby1.7
rubinius2.2
ruby2.1
ruby2.2

The interperter name is more or less the same except for rubinius. Their
binary is called rbx. So we get:

interpretername = "#{binaryname}#{major}.#{minor}"

jruby1.7
rbx2.2
ruby2.1
ruby2.2

As we wanted to support gems for more than one version, we also need to
express that in their package names now.

gempackage = "#{interpretername}-rubygem-#{gemname}#{gemsuffix}"

# gemsuffix is optional. we only need it for packages where we need more
# than one version

jruby1.7-rubygem-tzinfo
rbx2.2-rubygem-tzinfo
ruby2.1-rubygem-tzinfo
ruby2.2-rubygem-tzinfo

The gem binary naming is a bit longer now as well. But most of you
should not notice anything as update-alternatives covers this:

gembinary = “#{binaryname}.#{interpretername}-#{gemversion}”

$ ls -l /usr/bin/bundler*
lrwxrwxrwx 1 root root  25 25. Jul 19:04 /usr/bin/bundler -> /etc/alternatives/bundler
lrwxrwxrwx 1 root root  31 25. Jul 19:04 /usr/bin/bundler-1.6.5 -> /etc/alternatives/bundler-1.6.5
lrwxrwxrwx 1 root root  38  5. Sep 13:00 /usr/bin/bundler.rbx2.2 -> ../lib64/rubinius/gems/2.2/bin/bundler
lrwxrwxrwx 1 root root  33 25. Jul 19:04 /usr/bin/bundler.ruby2.1 -> /etc/alternatives/bundler.ruby2.1
-rwxr-xr-x 1 root root 504 25. Jul 18:53 /usr/bin/bundler.ruby2.1-1.6.5
lrwxrwxrwx 1 root root  33  8. Sep 19:19 /usr/bin/bundler.ruby2.2 -> /etc/alternatives/bundler.ruby2.2
-rwxr-xr-x 1 root root 504  5. Sep 15:57 /usr/bin/bundler.ruby2.2-1.6.5

Last but not least we have the ruby(abi). In the past it used to be just
the “version” number you had in your ruby paths. While this worked nicely for our packaging needs with just one ruby
interpreter. It is actually tricky for cases where a gem wants at least a certain ruby version. The gemspec
only has spec.required_ruby_version which is a version number. While in
MRI this number is compared with the ruby version number, in the case of
rubinius/jruby it is compared with the version of ruby language standard
that is implemented.

Provides:  jruby(abi) = 1.7
Provides:  rubinius(abi) = 2.1.0
Provides:  ruby(abi) = 2.1.0
Provides:  ruby(abi) = 2.2.0

One note: With rubinius this alone will not be enough as a requires,
we still have to add something like %requires_eq rubiniuspackage to
the sub package preamble. The reason is that rbx 2.2.10 and rbx 2.3.1
might both implement the ruby 2.1.0 api, but are not binary compatible
among each other.

One step at a time

Many of our gems nowadays build without having a buildrequires to their
runtime dependencies. We only generate the dependencies into the package
meta. Though some packages still could require e.g. rspec so they could
run their testsuite at build time. At this point generating multiple
spec files seemed like an easier solution: we would generate build deps
for each ruby interpreter/version into the spec file. But we were asked
not to, so we stick with a single spec file.

Fortunately rpm gives us macros, which on the other hand also need
support in the buildservice. Good thing is the guy to make that happen
for rpm and the OBS is the same person. :smiley:

BuildRequires:  %{ruby}
BuildRequires:  %{rubydevel}
BuildRequires:  %{rubygem somegem >= someversion}

Without going into too many details here, in home:darix:ruby this
expands to:

BuildRequires:  ruby2.1 ruby2.2 rubinius2.2
BuildRequires:  ruby2.1-devel ruby2.2-devel rubinius2.2-devel
BuildRequires:  rubygem(ruby:2.1.0:somegem) >= someversion rubygem(ruby:2.2.0:somegem) >= someversion rubygem(rubinius:2.2:somegem) >= someversion

The interested reader can check [2]. It involves recursive macro calls
and other fun things. You have been warned.

Most of our machinery for installing and cleaning up things was hidden
in macros/shell scripts already so adding a loop in those places was
trivial.

Suddenly we have multiple ruby interpreter/versions in the same build
root and our gems build with each of them.

So far nothing that would break all that horribly when we push it onto
d:l:r:e.

Rocks on the road

The problems started when we came to the %files sections in the spec
files. Until now we had been generating them into the spec file. That
means for every new ruby interpreter/major branch we would need to
regenerate all spec files.

That is not really a viable solution going forward. So we replaced the
static files section with a macro too.

%gem_packages

But this flexibility comes at a price. If building the gem fails for one
ruby interpreter/version, it will break the build for all. But you can
control which interpreters are even pulled into the build environment.

%define rb_build_versions %{rb_default_ruby}
BuildRequires: %{rubydevel}
BuildRequires: %{rubygem cheetah}

In home:darix:ruby this would expand to:

BuildRequires: ruby2.1-devel
BuildRequires: rubygem(ruby:2.1.0:cheetah)

The valid values for %rb_build_versions can be found on the in the
macros files of each interpreter package. We would have loved to use the
package names as the macro values but those values are passed into
macros again and macro names can not contain dots. For easier reading
you can look at the prjconf of home:darix:ruby. [2]

“Job done right?” “Well almost.”

There were all those little bits and pieces that creeped in now. We had
gems with %pre/%post scriptlets, because the gem was actually a somewhat
better tarball for a service. Rubygems also lacked a way to express
native buildrequires.

As it became clear that we will need to regenerate all the spec files in
d:l:r:e, we aimed for a solution that allowed us regenerating the spec
files at any time.

No more manually editing spec files

So we looked through a huge selection of spec files and checked what
things, we actually modified. Once that list was compiled, we created a
config file.

The config file is named gem2rpm.yml. Each field in the config file [3]
has a matching hook in the spec file template [4] or files section template
[5] (via %gem_packages).

gem2rpm.yml was patched to support the config file. Right now you have
to manually pass the config file, but there is a small shell wrapper [6]
that checks if the config file exists and adds the option automatically.

With that in place you can regenerate all spec files without worrying to
lose manual edits. There wont be any. And yes … in the development
process this has been done multiple times after fixes to the spec file
template.

The final result?

#
# spec file for package rubygem-tzinfo-0
<snip>
# This file was generated with a gem2rpm.yml and not just plain gem2rpm.
# All sections marked as MANUAL, license headers, summaries and descriptions
# can be maintained in that file. Please consult this file before editing any
# of those fields
#

Name:           rubygem-tzinfo-0
Version:        0.3.37
Release:        0
%define mod_name tzinfo
%define mod_full_name %{mod_name}-%{version}
%define mod_version_suffix -0
BuildRoot:      %{_tmppath}/%{name}-%{version}-build
BuildRequires:  ruby-macros >= 5
BuildRequires:  %{ruby}
BuildRequires:  %{rubygem gem2rpm}
BuildRequires:  %{rubygem rdoc > 3.10}
Url:            http://tzinfo.rubyforge.org/
Source:         http://rubygems.org/gems/%{mod_full_name}.gem
Source1:       gem2rpm.yml
Summary:        Daylight-savings aware timezone library
License:        MIT
Group:          Development/Languages/Ruby

%description
TZInfo is a Ruby library that uses the standard tz (Olson) database to provide
daylight savings aware transformations between times in different time zones.

%prep

%build

%install
%gem_install \
  --doc-files="CHANGES LICENSE README" \
  -f

%gem_packages

%changelog

As you see this spec uses a gem2rpm.yml, which looks like this:

---
:version_suffix: '-0'

With that spec file the build generates for me:

rbx2.2-rubygem-tzinfo-0-0.3.37-9.4.x86_64.rpm
rbx2.2-rubygem-tzinfo-doc-0-0.3.37-9.4.x86_64.rpm
rbx2.2-rubygem-tzinfo-testsuite-0-0.3.37-9.4.x86_64.rpm
ruby2.1-rubygem-tzinfo-0-0.3.37-9.4.x86_64.rpm
ruby2.1-rubygem-tzinfo-doc-0-0.3.37-9.4.x86_64.rpm
ruby2.1-rubygem-tzinfo-testsuite-0-0.3.37-9.4.x86_64.rpm
ruby2.2-rubygem-tzinfo-0-0.3.37-9.4.x86_64.rpm
ruby2.2-rubygem-tzinfo-doc-0-0.3.37-9.4.x86_64.rpm
ruby2.2-rubygem-tzinfo-testsuite-0-0.3.37-9.4.x86_64.rpm

This has already been used extensively to build
Discourse and
GitLab for SLE 12.

What is left?

If you look just at the things that are packaged using the new way, we
are done. But it would be nice to also support the versioning pattern
that we have used up to 13.1. Maybe even support building against
unversioned ruby versions. (Yes, in theory that is possible.)

Once we have taken those steps, we have to recreate all spec files in
d:l:r:e. Yes, you read correctly … recreate all spec files. While we
wrote all macros in a way to work as a drop in replacement, as soon as
we want to build for more than one ruby version, we need the
%gem_packages usage. I really hope enough people step up to the effort
so that the number of packages each person has to touch is kept in the
low double digits. Most of the work will be extracting the manual bits
into gem2rpm.yml files and then regenerate the spec file with the new
template. A more detailed mail will be sent later.

Last but not least we can improve the templates (move duplicated code
into gem2rpm e.g.)

Building non gem based ruby libraries is not covered by this new
packaging at all. For once many of the non rubygem based libraries are
bindings build within a larger build process. We would need to manually
do the loop for all ruby interpreter in those. I think in many cases we
only need those libraries for our system ruby. (like yast e.g.) For the
other cases we should work with upstream to switch the ruby bindings to
rubygems.

Also not covered yet, but it would be really useful: All the ruby
scripts shipped as part of the distro should use a shebang line pointing
to the versioned binary. You might ask “why? /usr/bin/ruby is a
hardlink!” While this is true … people can still replace it. Our
scripts/programs should not break in that situation. E.g. yast is
very critical in that regard. Again a gem based distribution makes it
easier as rubygems does it for us in that case. But it shouldn’t be much
work to add a small script to fix shebang lines and an rpmlint check
that finds all shebang lines that are unversioned.