- Introduction
- Getting rid of RVM, rbenv and bundler
- GNU Guix installation
- Ruby
- Adding system shared gems
- Adding Gems in user land ($HOME)
- Adding a Ruby gem to Guix
- Generating GNU Guix Gem packages
- More on creating Ruby packages
- Troubleshooting
GNU Guix, or life without bundler, rvm etc. Yes, you can still use these tools but there really is no longer a need because GNU Guix supports native Ruby gems!
The GNU software packaging project is packaging done right. Ruby deployment with rubygems used to be pretty good, years ago, but over time it has turned into a nightmare of dependencies. Tools like RVM, rbenv and bundler try to bring some level of control for running multiple versions of Ruby and gems, but over the last years they are showing their limitations clearly. Every time I needed to work with RVM I started throwing things at my computer.
So when should you look at GNU Guix? You should look at Guix when you
- are serious about software deployment
- need to handle multiple versions of Ruby or gems
- want clear isolation of dependencies
- want clean separation of gems
- want a reproducible environment
Just a teaser: run Ruby with its dependencies inside a Guix container that is completely isolated from the rest of the system:
guix environment -C guix --network --ad-hoc ruby ruby-sinatra
GNU Guix allows you to define a software package once with all its dependencies. Every time you install the package it gets reproduced exactly with its exact dependency graph, all the way down to glibc. See this figure. Whether you are a sysadmin who needs to deploy an exact Rails stack or you are a developer and need to support user environment, GNU Guix is the solution you require. Use cases are
- test gems using multiple versions of Ruby
- install concurrent rubies with or without linked openssl support
- run minimal ruby to be exposed to the web
- update ruby in production and roll-back after a problem
- run multiple versions of the same gem against one ruby
- run multiple versions of openssl dependencies
Use your imagination. The point is that you control the full dependency graph. Always. You can even give users rights to install and share software because the underlying system is ‘immutable’. Existing graphs can not be overwritten by others.
GNU Guix is a next generation software package installer with a range of features, including sane dependency handling, transactional and reproducible installs which can be rolled back. In short, GNU Guix has resolved the fundamental problems of software deployment and management. GNU Guix also should play well with Docker and VMs.
GNU Guix is getting mature with almost a thousand software packages. In this document I explain what the philosophy is of Ruby (gem) software management and how we put it together. Feel free to ask questions and contribute ideas. There are multiple possible tactics for sane dependency handling that GNU Guix could support.
In addition GNU Guix has its own build farm and continuous integration for submitted guix expressions (packages). We’ll also add GNU Guix support to Travis CI soon.
And, as a thrown in benefit, running tests with cucumber is also much faster without bundler wrapping!
Ruby gems are natively supported from GNU Guix. This means we can get rid of the RVM-rbenv-bundler trinity. All we use here is the ruby interpreter with the accompanying gem tool. An example of a patch in which I removed bundler and jeweler from a gem can be found here. Note that you can still use bundler, if you want, but I no longer have a use for it (note that Trevor-CI still uses the gemspec with bundler for the tests).
The Achilles heel of introducing GNU Guix at this point is that it is not standard in Linux distributions. That support will come because GNU Guix is the official packaging system of the GNU project and is orthogonal to existing software packaging systems (everything is installed in /gnu). It will happen. And even without distribution support, guix is trivial to install and easy to support on any Linux system (and soon other targets).
Currently there are multiple ways of installing GNU Guix. See the INSTALL document in this repository. There is a binary installer (i.e. unpack tarball in root) which should make things easier. Note that if you have Guix installed and running from /guix/store you can just transfer the setup to another machine! Make sure to copy the database in /var too. The only actual dependency is the Linux kernel (not even glibc)! To add packages see HACKING.
The basic idea of GNU Guix is simple. A HASH value (SHA256) is calculated over the inputs to a build. This includes the source code of Ruby, and the switches used over configure and make. The software is installed under the HASH, for example I have Ruby 2.1.2 and 2.1.3 on my system sitting under
/gnu/store/wy8hwm8c01r2lsgkci67amg66pk9ml7a-ruby-2.1.3:
bin include lib share
/gnu/store/yb9z2y7ndzra9r3x7l3020zjpds43yyc-ruby-2.1.2:
bin include lib shareand, for example, another ruby 1.8.7 under
/gnu/store/2sd245py3i04y4yapvnp8cdpsykijllh-ruby-1.8.7:
bin lib shareThey are cleanly separated. Now if I were to change the configure for 2.1.3, for example a build without openssl, it would simply become another HASH and therefore directory.
It gets even better, the HASH value is also calculated over the dependencies. So, if you are running two different glibc’s on your system (each under its own HASH directory), or openssl’s, the ruby interpreter gets build against one of each and calculates a unique HASH. So you can theoretically have four concurrent Ruby 2.1.3 installations, compiled against any combination of two glibc’s and two openssl’s. The point, again, is that you have full control over the dependency graph!
To make a Ruby visible to a user, GNU Guix uses symlinks. Installing a particular Ruby will symlink a so-called profile in ~/.guix-profile/bin. To run Ruby, simply run it as
~/.guix-profile/bin/ruby -v
ruby 2.1.3p242 (2014-09-19 revision 47630) [x86_64-linux]The libraries that come with Ruby are also symlinked via ~/.guix-profile/lib/ruby/2.1.0/. The numbering does not matter too much since it points to an immutable (read-only) directory in
~/.guix-profile/lib -> /gnu/store/ziy7a6zib846426kprc7fgimggh8bz97-ruby-2.1.3/libWhen a profile is built, the symlinks are always placed as close to the root as possible. In the case above, the ruby package is the only thing in the profile that has a top-level ‘lib’ directory. If later another package gets installed that contains ‘lib’, the newly-built profile will create ‘lib’ as a directory and move the symlinks further down.
This means that you can access Ruby libraries shipped with a particular Ruby version, but that you can not write new files into that directory! The Ruby installation is carved in stone.
To check the version of openssl you can do
~/.guix-profile/bin/ruby -ropenssl -e "puts OpenSSL::VERSION"
To list the shared libraries:
ldd ~/.guix-profile/bin/ruby
linux-vdso.so.1 (0x00007ffee8533000)
libpthread.so.0 => /gnu/store/hy2hi0zj5hrqkmkhpdxf04c9bcnlnsf9-glibc-2.21/lib/libpthread.so.0 (0x00007efe20b58000)
libdl.so.2 => /gnu/store/hy2hi0zj5hrqkmkhpdxf04c9bcnlnsf9-glibc-2.21/lib/libdl.so.2 (0x00007efe20954000)
libcrypt.so.1 => /gnu/store/hy2hi0zj5hrqkmkhpdxf04c9bcnlnsf9-glibc-2.21/lib/libcrypt.so.1 (0x00007efe2071d000)
libm.so.6 => /gnu/store/hy2hi0zj5hrqkmkhpdxf04c9bcnlnsf9-glibc-2.21/lib/libm.so.6 (0x00007efe2041b000)
libgcc_s.so.1 => /gnu/store/rsw0dkmv1x2krv9pl1ciai1h235r9nb7-gcc-4.8.4-lib/lib/libgcc_s.so.1 (0x00007efe20205000)
libc.so.6 => /gnu/store/hy2hi0zj5hrqkmkhpdxf04c9bcnlnsf9-glibc-2.21/lib/libc.so.6 (0x00007efe1fe65000)
/gnu/store/hy2hi0zj5hrqkmkhpdxf04c9bcnlnsf9-glibc-2.21/lib/ld-linux-x86-64.so.2 (0x00007efe20d75000)System shared gems are GNU Guix packages (unless you start explicitly overriding above GEM_PATHs). The advantage of using GNU Guix is that the dependency graph is explicit and people can easily share installations. A gem gets installed with its version under its own HASH dir, e.g.
/gnu/store/HASH-rspec-1.0.0This means (again) you can support multiple versions of gems. Under GNU Guix gems become first-rate citizens in a software stack.
To install nokogiri:
guix package -i ruby-nokogiri
or, if running in the source directory
./pre-inst-env guix package -i ruby-nokogiri
Note that this install dependencies ruby, libxml2 and libxslt if not already installed!
The following files will be downloaded:
/gnu/store/7vbz3h82hh11wmaxfvxswsld24ljwhz9-ruby-nokogiri-1.6.6.2
/gnu/store/grd4vpgp6cbxfcwmp5n1gssv8svpgrvf-ruby-mini-portile-0.6.2
/gnu/store/476b4vab2x5ryccwfhy839v5c6vmz59x-glibc-utf8-locales-2.21
/gnu/store/2x8w06phz69hq7yr457xy0n46vws0wpl-texinfo-6.0
/gnu/store/b16xqps0fxgkx5ffw7r549h1gy53rj63-gzip-1.6
/gnu/store/c158g4fki606z1g0l240kknprfwdls0a-coreutils-8.24
/gnu/store/f033flfhq0qlzxpicbmq8b4x09y4h148-ncurses-5.9
/gnu/store/gjs5zk5366a4bdwyy6vv1x8cfx7b092m-perl-5.16.1
/gnu/store/6gkslyn4iprga0w78d57g3dzsks38sia-libxslt-1.1.28
/gnu/store/famqzp3sb1mldklv6m18r4v8nq0baf2j-libxml2-2.9.2
/gnu/store/ippi1rw3869rzv21v3ixvzrim40r2s02-ruby-2.2.3So, unlike with rubygems on its own, this is the full dependency stack. Perl is installed because it is a build dependency somewhere in the build system. Likewise, if you install ruby-pg, postgresql and dependencies will also get installed.
Note that guix installs in ~/.guix-profile/lib/ruby/gems/2.2.0/gems/ (well, actually symlinks). The version numbering 2.2.0 follows the gem convention to share gems on major numbers. Updating a minor version number will keep the gems. If this is not what you want (i.e. true version isolation) make sure to use guix profiles for individual Rubies and use only that GEM_PATH.
GNU Guix Ruby comes with gem support out of the box. The gem tool also is symlinked in ~/.guix-profile/bin. When we run `gem env’ it says
export PATH=~/.guix-profile/bin/:$PATH
gem env
RubyGems Environment:
- RUBYGEMS VERSION: 2.2.2
- RUBY VERSION: 2.1.3 (2014-09-19 patchlevel 242) [x86_64-linux]
- INSTALLATION DIRECTORY: /gnu/store/ziy7a6zib846426kprc7fgimggh8bz97-ruby-2.1.3/lib/ruby/gems/2.1.0
- RUBY EXECUTABLE: /gnu/store/ziy7a6zib846426kprc7fgimggh8bz97-ruby-2.1.3/bin/ruby
- EXECUTABLE DIRECTORY: /gnu/store/ziy7a6zib846426kprc7fgimggh8bz97-ruby-2.1.3/bin
- SPEC CACHE DIRECTORY: /home/user/.gem/ziy7a6zib846426kprc7fgimggh8bz97-ruby-2.1.3/specs
- RUBYGEMS PLATFORMS:
- ruby
- x86_64-linux
- GEM PATHS:
- /gnu/store/ziy7a6zib846426kprc7fgimggh8bz97-ruby-2.1.3/lib/ruby/gems/2.1.0
- /home/user/.gem/ziy7a6zib846426kprc7fgimggh8bz97-ruby-2.1.3/2.1.0The general idea here is that we allow users to install their own gems, but cleanly separated against the HASH dir that comes with the Ruby installation. This way there is clear isolation between different installed versions of Ruby. Unlike RVM and rbenv, there is NO (accidental) sharing between different Ruby installations!
To achieve clean separation we can patch Ruby and gem to make use of the new GEM_PATHs or we can create a wrapper script which presets the PATH. At this point I favour the patching because Ruby gem has these paths built-in. In practice we use a script to modify the environment. I wrote a bash script which does this can be found as ./scripts/ruby-guix-env (more on that below).
When you do a grep on the files in the Ruby installation dir, all references to GEM_HOME and GEM_PATH occur in files under lib/ruby/2.1.0/rubygems/.
When you override these with
env GEM_HOME=gem_home GEM_PATH=gem_path GEM_SPEC_CACHE=gem_spec_cache gem env
RubyGems Environment:
- RUBYGEMS VERSION: 2.2.2
- RUBY VERSION: 2.1.3 (2014-09-19 patchlevel 242) [x86_64-linux]
- INSTALLATION DIRECTORY: gem_home
- RUBY EXECUTABLE: /gnu/store/ziy7a6zib846426kprc7fgimggh8bz97-ruby-2.1.3/bin/ruby
- EXECUTABLE DIRECTORY: gem_home/bin
- SPEC CACHE DIRECTORY: gem_spec_cache
- RUBYGEMS PLATFORMS:
- ruby
- x86_64-linux
- GEM PATHS:
- gem_home
- gem_pathyou can see Rubygems cleanly honours these environment variables (which is what, for example, rbenv utilises). Clean separation can thus be enforced from the command line with
export GEM_PATH=/home/pjotrp/.gem/ziy7a6zib846426kprc7fgimggh8bz97-ruby-2.1.3/2.1.0
export GEM_HOME=$GEM_PATH
export GEM_SPEC_CACHE=/home/pjotrp/.gem/ziy7a6zib846426kprc7fgimggh8bz97-ruby-2.1.3/specs
mkdir -p $GEM_PATH
mkdir -p $GEM_SPEC_CACHE
gem envNow local gem installs should work, e.g.
gem install -V bundler
gem install -V bio-loggerand
gem list -dwill tell you where the gems are installed. To use bundler you can call
~/.gem/ziy7a6zib846426kprc7fgimggh8bz97-ruby-2.1.3/2.1.0/bin/bundlerThe paths may look a bit long, but that guarantees separation! The PATH should be set to
export PATH=$HOME/.guix-profile/bin:$HOME/.gem/ziy7a6zib846426kprc7fgimggh8bz97-ruby-2.1.3/2.1.0/binand run
bundle
bundle exec rakeWhen there is a problem with your gems, simply clean up $HOME/.gem/ziy7a6zib846426kprc7fgimggh8bz97-ruby-2.1.3 and start from scratch with a clean Ruby installation. Or, more rigorously, start writing system shared gems.
I wrote a bash script which does this can be found as ./scripts/ruby-guix-env and can be run as
source ./scripts/ruby-guix-envGems build in GEM_HOME may look for linked libraries
export LD_LIBRARY_PATH=$LD_LIBRARY_PATH:$HOME/.guix-profile/lib
Recently bundler support was added! After installing bundler you may want to set the GEM_PATH to include the guix-profile gem location, e.g.,
export GEM_PATH=/home/pjotrp/.gem/x4z4vi0aynd5krn4fz3l7ix9187z0g8y-ruby-2.2.2/2.1.0:$HOME/.guix-profile/lib/ruby/gems/2.2.0
Check with `gem env’ and try to `run bundle’.
Bundle may have trouble building native extension. For this see the writeup in RUBY-NOKOGIRI.org
So, now you ask, ‘Now, HOW do I add a Guix gem’? The good news is that it is easy because Guix can install Ruby gems natively. Still, a package description (guix expression) is required to have the gem added to the GNU Guix distribution. Once it is there in git, it will get built on the build farm (continuous integration) and be available in binary form for all GNU Guix users!
Find your gem on http://rubygems.org/ and download the gem to get the HASH value:
guix download https://rubygems.org/downloads/bio-locus-0.0.6.gem /gnu/store/5ddsb4k6g9pn66klfw1d42jb90yz2iqf-bio-locus-0.0.6.gem 0l303w5kzsriqs5gvcbgx5l236hajj5bf76fpv1yymiwnjp7d97k
Now we add the following package to guix/gnu/ruby.scm (it may make sense to create a new git branch) using guidelines.
cd guix git checkout -b bio-locus
(define-public ruby-bio-locus
(package
(name "ruby-bio-locus")
(version "0.0.6")
(source
(origin
(method url-fetch)
(uri (rubygems-uri "bio-locus" version))
(sha256
(base32
"0l303w5kzsriqs5gvcbgx5l236hajj5bf76fpv1yymiwnjp7d97k"))))
(build-system ruby-build-system)
(arguments
'(#:tests? #t)) ; no tests
(synopsis "Bio-locus is a tool for fast querying of genome locations")
(description "This tabix-like tool essentially allows your to
store this chr+pos or chr+pos+alt information in a fast database.")
(home-page "https://github.com/pjotrp/bio-locus")
(license license:expat)))Note the HASH is the same as the one we got with guix download.
Actually, guix can also give you an example expression for a gem with
./pre-inst-env guix import gem bio-locus
(package
(name "ruby-bio-locus")
(version "0.0.7")
(source
(origin
(method url-fetch)
(uri (rubygems-uri "bio-locus" version))
(sha256
(base32
"02vmrxyimkj9sahsp4zhfhnmbvz6dbbqz1y01vglf8cbwvkajfl0"))))
(build-system ruby-build-system)
(synopsis
"A tool for fast querying and filtering of genome locations in VCF and other formats")
(description
"A tool for fast querying and filtering of genome locations in VCF and other formats")
(home-page "http://github.com/pjotrp/bio-locus")
(license expat))which can help to get started!
Install the package with
./pre-inst-env guix package -i ruby-bio-locus (...) LoadError: cannot load such file -- bundler
Oops, it gave an error. That is because there is a bundler dependency for some reason. We can add the dependency or update the gem to remove it. The great thing is that guix builds packages in isolation - missing build or runtime dependencies are always caught. So we add
(native-inputs
`(("bundler" ,bundler)))That would work, but I ended up updating the gem because there was some more stuff to remove. The final version is pretty clean.
Hint: use the -K switch if you want to keep the unpacked build repository to see where the error occurred.
First check the syntax:
./pre-inst-env guix lint ruby-bio-locus
Next make a single patch following the guidelines and submit it to the mailing list after
git format-patch -1
Take a gem name and try
guix import gem bioruby
If you are hungry for more internals, see the file RUBY-NOKOGIRI.org for more information.
When you get
ERROR: Could not find a valid gem 'cucumber' (>= 0), here is why:
Unable to download data from https://rubygems.org/ - SSL_connect returned=1 errno=0 state=error: certificate verify failed (https://api.rubygems.org/specs.4.8.gz)
set
export SSL_CERT_FILE=/etc/ssl/certs/ca-certificates.crt