Rails cannot write files to ./tmp with jammy full builder

[benydc]

When using Rails framework, the ActiveRecord needs to write caches or other files in the root project tmp folder (/workspace/tmp) but it fails with permission denied to write files to that folder. Rails Framework offers no config to change this to another tmp path, and some gem libraries for Rails have this hardcoded also.

Expected Behavior

CNB to write files in (/workspace/tmp)

Current Behavior

Access denied for CNB to write in /workspace/tmp
Example error message: Errno::EACCES · Permission denied @ dir_s_mkdir - /workspace/tmp/1670939080-843440521812643-0007-6864

Possible Solution

Allow writes to /workspace/tmp for CNB User/Rails Framework

Steps to Reproduce

1. Setup rails project using https://guides.rubyonrails.org/getting_started.html
2. Set default builder pack config default-builder paketobuildpacks/builder-jammy-full
3. Root project we run pack build rails-paketo-jammy
4. Run container docker run -e RAILS_MASTER_KEY="<master-key>" rails-paketo-jammy (master key value is at ./config/master.key)
5. Observe that it fails to run puma with error: Permission denied @ rb_sysopen - tmp/pids/server.pid

Motivations

We have switched to Jammy because our backend requires a system library version that is not supported in Bionic.

[robdimsdale]

Thanks for opening this @benydc .

I am able to reproduce this, and I've validated that this isn't an issue on the Bionic stack. It can also be reproduced with the Paketo samples application for rails assets.

The underlying issue is that the Jammy stack has different users for build and run time. This is by design - it is a security feature - but it does occasionally have unintended consequences, like this. This feature was introduced with the Jammy stack, and we are unlikely to revert that decision. Instead we'll need to ensure the buildpacks work with that architecture.

We will need to investigate what changes we need to make to the buildpacks to ensure out-of-the-box support for Rails apps on Jammy. For now, however, the easiest workaround is to relax the permissions of files/directories that need to be modified at run-time. I found that I had modify ./tmp/pids and also ./Gemfile.lock.

chmod 777 ./tmp/pids
chmod 777 ./Gemfile.lock

After doing this, here is an example output of running the application and making a request from the host machine via curl localhost:9292:

❯ docker run -it --rm -e RAILS_MASTER_KEY=$(cat config/master.key) -p9292:9292 rails-paketo-jammy
WARNING: The requested image's platform (linux/amd64) does not match the detected host platform (linux/arm64/v8) and no specific platform was requested
Puma starting in single mode...
* Puma version: 5.6.5 (ruby 3.1.2-p20) ("Birdie's Version")
*  Min threads: 5
*  Max threads: 5
*  Environment: production
*          PID: 1
W, [2022-12-19T14:19:49.211633 #1]  WARN -- : You are running SQLite in production, this is generally not recommended. You can disable this warning by setting "config.active_record.sqlite3_production_warning=false".
* Listening on http://0.0.0.0:9292
Use Ctrl-C to stop
I, [2022-12-19T14:19:52.328206 #1]  INFO -- : [cd04f926-da8f-46e8-9e25-d2e333dcb9f1] Started GET "/" for 172.17.0.1 at 2022-12-19 14:19:52 +0000
I, [2022-12-19T14:19:52.334255 #1]  INFO -- : [cd04f926-da8f-46e8-9e25-d2e333dcb9f1] Processing by ArticlesController#index as */*
I, [2022-12-19T14:19:52.351382 #1]  INFO -- : [cd04f926-da8f-46e8-9e25-d2e333dcb9f1]   Rendered articles/index.html.erb within layouts/application (Duration: 2.2ms | Allocations: 177)
I, [2022-12-19T14:19:52.413314 #1]  INFO -- : [cd04f926-da8f-46e8-9e25-d2e333dcb9f1]   Rendered layout layouts/application.html.erb (Duration: 67.4ms | Allocations: 2692)
I, [2022-12-19T14:19:52.415358 #1]  INFO -- : [cd04f926-da8f-46e8-9e25-d2e333dcb9f1] Completed 200 OK in 80ms (Views: 73.0ms | Allocations: 4091)

I recognize that setting global modify permissions on files and directories isn't ideal, so I'd like to keep this issue open to drive out a better solution.

@paketo-buildpacks/ruby-maintainers let's use this issue to drive out out-of-the-box support for Jammy on an example rails app.

As an aside @benydc, there's no need to justify switching to Jammy (22.04). Bionic (20.04) will be out of support upstream around April 2023 so we're encouraging folks to switch over to Jammy as soon as they're able. Thank you for trying it out so quickly after we released support for Jammy!

[benydc]

@robdimsdale thank you! I will use the run-time image to relax the permissions on ./tmp folder.

[robdimsdale]

@benydc are you using a custom run image? If not, I don't think you'll be able to relax the permissions at run-time, because the run-time user doesn't have the permissions to do this (that's the entire point of this issue).

I ran the chmod 777 ... commands locally, and I'd expect that you would have to commit those updated permissions in order for the app to build on other machines.

[benydc]

@robdimsdale yes, I'm using custom run image, in there I will be able to switch to root user and apply the chmod 777 ./tmp and should work?

[sophiewigmore]

we also saw this with one of our Rails sample apps. Another solution that may work for you:
If you have the following in your config/puma.rb

pidfile ENV.fetch("PIDFILE") { "tmp/pids/server.pid" }

Upgrade it to:

pidfile ENV.fetch("PIDFILE") { "/tmp/pumd.pid" }

[robdimsdale]

@benydc

@robdimsdale yes, I'm using custom run image, in there I will be able to switch to root user and apply the chmod 777 ./tmp and should work?

Yes, I'd expect that should work fine.

@sophiewigmore - good catch! I didn't go investigating that angle because I also ran into issues with the permissions on the Gemfile.lock - have you see that issue crop up elsewhere too?

[sophiewigmore]

Yea I saw this when we started added language family tests against Jammy. #876

I didn't think about filing an issue though.

[ryanmoran]

What's going on with the Gemfile.lock being modified at runtime? That seems pretty odd.

[robdimsdale]

What's going on with the Gemfile.lock being modified at runtime? That seems pretty odd.

@ryanmoran agreed! I didn't dig into this further. I just followed the Rails getting started guide and I saw that the Gemfile.lock couldn't be written at runtime. However, I can't reproduce it now.

[thirdeyenick]

We also ran into this issue with the Ubuntu Jammy builder. As the runtime user and the buildtime user share the same unix group ID (just the user ID is different) wouldn't it be possible to make certain directories writable beneath /workspace by allowing the group write access to it? I am not sure which exact buildpack would be responsible for this though.

[tcdowney]

We're running into this issue when trying to run our Postfacto retro board app. It fails with the following:

 2023-07-20T21:52:22.57-0600 [APP/] OUT => Booting Puma
   2023-07-20T21:52:22.57-0600 [APP/] OUT => Rails 7.0.4.3 application starting in production
   2023-07-20T21:52:22.57-0600 [APP/] OUT => Run `bin/rails server --help` for more startup options
   2023-07-20T21:52:22.58-0600 [APP/] OUT /layers/paketo-buildpacks_mri/mri/lib/ruby/3.2.0/fileutils.rb:406:in `mkdir': Permission denied @ dir_s_mkdir - /workspace/tmp/pids (Errno::EACCES)
   2023-07-20T21:52:22.58-0600 [APP/] OUT 	from /layers/paketo-buildpacks_mri/mri/lib/ruby/3.2.0/fileutils.rb:406:in `fu_mkdir'
   2023-07-20T21:52:22.58-0600 [APP/] OUT 	from /layers/paketo-buildpacks_mri/mri/lib/ruby/3.2.0/fileutils.rb:384:in `block (2 levels) in mkdir_p'
   2023-07-20T21:52:22.58-0600 [APP/] OUT 	from /layers/paketo-buildpacks_mri/mri/lib/ruby/3.2.0/fileutils.rb:382:in `reverse_each'
   2023-07-20T21:52:22.58-0600 [APP/] OUT 	from /layers/paketo-buildpacks_mri/mri/lib/ruby/3.2.0/fileutils.rb:382:in `block in mkdir_p'
   2023-07-20T21:52:22.58-0600 [APP/] OUT 	from /layers/paketo-buildpacks_mri/mri/lib/ruby/3.2.0/fileutils.rb:374:in `each'
   2023-07-20T21:52:22.58-0600 [APP/] OUT 	from /layers/paketo-buildpacks_mri/mri/lib/ruby/3.2.0/fileutils.rb:374:in `mkdir_p'
   2023-07-20T21:52:22.58-0600 [APP/] OUT 	from /layers/paketo-buildpacks_bundle-install/launch-gems/ruby/3.2.0/gems/railties-7.0.4.3/lib/rails/commands/server/server_command.rb:71:in `block in create_tmp_directories'
   2023-07-20T21:52:22.58-0600 [APP/] OUT 	from /layers/paketo-buildpacks_bundle-install/launch-gems/ruby/3.2.0/gems/railties-7.0.4.3/lib/rails/commands/server/server_command.rb:70:in `each'
   2023-07-20T21:52:22.58-0600 [APP/] OUT 	from /layers/paketo-buildpacks_bundle-install/launch-gems/ruby/3.2.0/gems/railties-7.0.4.3/lib/rails/commands/server/server_command.rb:70:in `create_tmp_directories'
   2023-07-20T21:52:22.58-0600 [APP/] OUT 	from /layers/paketo-buildpacks_bundle-install/launch-gems/ruby/3.2.0/gems/railties-7.0.4.3/lib/rails/commands/server/server_command.rb:34:in `start'
   2023-07-20T21:52:22.58-0600 [APP/] OUT 	from /layers/paketo-buildpacks_bundle-install/launch-gems/ruby/3.2.0/gems/railties-7.0.4.3/lib/rails/commands/server/server_command.rb:143:in `block in perform'
   2023-07-20T21:52:22.58-0600 [APP/] OUT 	from <internal:kernel>:90:in `tap'
   2023-07-20T21:52:22.58-0600 [APP/] OUT 	from /layers/paketo-buildpacks_bundle-install/launch-gems/ruby/3.2.0/gems/railties-7.0.4.3/lib/rails/commands/server/server_command.rb:134:in `perform'

<truncated>

Which ends up pointing at https://github.com/rails/rails/blob/main/railties/lib/rails/commands/server/server_command.rb#L71-L73 -- not something we can change.

Since the Bionic stack is no longer supported I feel there has to be a better solution than requiring users to make their own run images. @robdimsdale is that still the current recommendation?

[tcdowney]

For what it's worth, I'm thinking of trying out an approach similar to this chmod buildpack as a workaround. Unless you have better ideas @dmikusa ?

[robdimsdale]

@tcdowney I agree. We should support rails apps out of the box with the jammy stack. Asking users to maintain custom run images is not a viable path.

The chmod buildpack might help, but could you also do something like the following:

mkdir -p ./tmp
touch ./tmp/.gitkeep
chmod 777 ./tmp

Maybe also with:

mkdir -p ./tmp/pids
touch ./tmp/pids/.gitkeep
chmod 777 ./tmp/pids

I haven't tried this out so I don't know if I'm missing something. But it seems like that would be a quick thing to try while we iterate on better support in the buildpack itself.

[tcdowney]

Hmm @robdimsdale I may have misunderstood what you said, but making those changes in the app repo prior to pushing doesn't seem to have an effect. I ended up forking that chmod buildpack and tailoring it to only chmod the ./tmp dir. That seems to work fine for our use case.

[robdimsdale]

Hmm @robdimsdale I may have misunderstood what you said, but making those changes in the app repo prior to pushing doesn't seem to have an effect. I ended up forking that chmod buildpack and tailoring it to only chmod the ./tmp dir. That seems to work fine for our use case.

You probably did it correctly. I wasn't sure if that would work or not. Thanks for trying it out.

I'm glad you're unblocked, and I also think this is something we should fix at the buildpack level.

[carlpartridge]

Any updates on this? After trying everything else in this thread we were able to work around this using the chmod-buildpack as well (as a basic starting point, as it seems fairly outdated at this point).

[sophiewigmore]

Hey @carlpartridge, no update, it totally fell off our radar. I think we definitely need to investigate this ASAP since Bionic is EOL. Thanks for the poke, I'll try to prioritize some time next week to look into a fix!

[sophiewigmore]

@carlpartridge end of year craziness / holiday time we likely won't get to this for a bit unfortunately. Contributions are welcome! It is on my radar though

[matthewford]

@sophiewigmore Hi is there an update on this? We're migrating our Rails app to try to use this buildpack.

[taraszka]

@sophiewigmore Hi is there an update on this? We're migrating our Rails app to try to use this buildpack.

add to your repo project.toml:

[_]
schema-version = "0.2"

[[io.buildpacks.post.group]]
uri = "docker://miget/rails-chmod-buildpack"

works for us.

[sophiewigmore]

Hi friends - I'm sorry there's been no movement on our end on this issue, its been hard to fit things in with shifting priorities. @taraszka thank you for the workaround!