Skip to content

Latest commit

 

History

History
1030 lines (839 loc) · 35.3 KB

README.md

File metadata and controls

1030 lines (839 loc) · 35.3 KB

Shrine is a toolkit for file attachments in Ruby applications. Some highlights:

If you're curious how it compares to other file attachment libraries, see the Advantages of Shrine.

Resources

Resource URL
Website shrinerb.com
Demo code Roda / Rails
Source github.com/shrinerb/shrine
Wiki github.com/shrinerb/shrine/wiki
Bugs github.com/shrinerb/shrine/issues
Help & Discussion groups.google.com/group/ruby-shrine

Contents

Quick start

Add Shrine to the Gemfile and write an initializer which sets up the storage and loads the ORM plugin:

# Gemfile
gem "shrine", "~> 2.0"
require "shrine"
require "shrine/storage/file_system"

Shrine.storages = {
  cache: Shrine::Storage::FileSystem.new("public", prefix: "uploads/cache"), # temporary
  store: Shrine::Storage::FileSystem.new("public", prefix: "uploads"),       # permanent
}

Shrine.plugin :sequel # or :activerecord
Shrine.plugin :cached_attachment_data # for retaining the cached file across form redisplays
Shrine.plugin :restore_cached_data # re-extract metadata when attaching a cached file
Shrine.plugin :rack_file # for non-Rails apps

Next decide how you will name the attachment attribute on your model, and run a migration that adds an <attachment>_data text or JSON column, which Shrine will use to store all information about the attachment:

Sequel.migration do
  change do
    add_column :photos, :image_data, :text
  end
end

In Rails with Active Record the migration would look similar:

$ rails generate migration add_image_data_to_photos image_data:text
class AddImageDataToPhotos < ActiveRecord::Migration
  def change
    add_column :photos, :image_data, :text
  end
end

Now you can create an uploader class for the type of files you want to upload, and add a virtual attribute for handling attachments using this uploader to your model. If you do not care about adding plugins or additional processing, you can use Shrine::Attachment.

class ImageUploader < Shrine
  # plugins and uploading logic
end
class Photo < Sequel::Model # ActiveRecord::Base
  include ImageUploader::Attachment.new(:image) # adds an `image` virtual attribute
end

Let's now add the form fields which will use this virtual attribute. We need (1) a file field for choosing files, and (2) a hidden field for retaining the uploaded file in case of validation errors and for potential direct uploads.

# with Rails form builder:
form_for @photo do |f|
  f.hidden_field :image, value: @photo.cached_image_data
  f.file_field :image
  f.submit
end
# with Simple Form:
simple_form_for @photo do |f|
  f.input :image, as: :hidden, input_html: { value: @photo.cached_image_data }
  f.input :image, as: :file
  f.button :submit
end
# with Forme:
form @photo, action: "/photos", enctype: "multipart/form-data" do |f|
  f.input :image, type: :hidden, value: @photo.cached_image_data
  f.input :image, type: :file
  f.button "Create"
end

Note that the file field needs to go after the hidden field, so that selecting a new file can always override the cached file in the hidden field. Also notice the enctype="multipart/form-data" HTML attribute, which is required for submitting files through the form (the Rails form builder will automatically generate this for you).

When the form is submitted, in your router/controller you can assign the file from request params to the attachment attribute on the model.

# In Rails:
class PhotosController < ApplicationController
  def create
    Photo.create(photo_params)
    # ...
  end

  private

  def photo_params
    params.require(:photo).permit(:image)
  end
end
# In Sinatra:
post "/photos" do
  Photo.create(params[:photo])
  # ...
end

Once a file is uploaded and attached to the record, you can retrieve a URL to the uploaded file with #<attachment>_url and display it on the page:

<!-- In Rails: -->
<%= image_tag @photo.image_url %>
<!-- In HTML: -->
<img src="<%= @photo.image_url %>" />

Storage

A "storage" in Shrine is an object that encapsulates communication with a specific storage service, by implementing a common public interface. Storage instances are registered under an identifier in Shrine.storages, so that they can later be used by uploaders.

Previously we've shown the FileSystem storage which saves files to disk, but Shrine also ships with S3 storage which stores files on AWS S3 (or any S3-compatible service such as DigitalOcean Spaces or MinIO).

# Gemfile
gem "aws-sdk-s3", "~> 1.14" # for AWS S3 storage
require "shrine/storage/s3"

s3_options = {
  bucket:            "<YOUR BUCKET>", # required
  access_key_id:     "<YOUR ACCESS KEY ID>",
  secret_access_key: "<YOUR SECRET ACCESS KEY>",
  region:            "<YOUR REGION>",
}

Shrine.storages = {
  cache: Shrine::Storage::S3.new(prefix: "cache", **s3_options),
  store: Shrine::Storage::S3.new(**s3_options),
}

The above example sets up S3 for both temporary and permanent storage, which is suitable for direct uploads. The :cache and :store names are special only in terms that the attacher will automatically pick them up, you can also register more storage objects under different names.

See the FileSystem and S3 storage docs for more details. There are many more Shrine storages provided by external gems, and you can also create your own storage.

Uploader

Uploaders are subclasses of Shrine, and they wrap the actual upload to the storage. They perform common tasks around upload that aren't related to a particular storage.

class MyUploader < Shrine
  # image attachent logic
end

It's common to create an uploader for each type of file that you want to handle (ImageUploader, VideoUploader, AudioUploader etc), but really you can organize them in any way you like.

Uploading

The main method of the uploader is #upload, which takes an IO-like object and a storage identifier on the input, and returns a representation of the uploaded file on the output.

MyUploader.upload(file, :store) #=> #<Shrine::UploadedFile>

Internally this instantiates the uploader with the storage and calls #upload on it:

uploader = MyUploader.new(:store)
uploader.upload(file) #=> #<Shrine::UploadedFile>

Some of the tasks performed by #upload include:

The second argument is a context hash which is forwarded to places like metadata extraction and location generation, but it has a few special options:

uploader.upload(io, metadata: { "foo" => "bar" })           # add metadata
uploader.upload(io, location: "path/to/file")               # specify custom location
uploader.upload(io, upload_options: { acl: "public-read" }) # add options to Storage#upload

IO abstraction

Shrine is able to upload any IO-like object that implement methods #read, #rewind, #eof? and #close whose behaviour matches the IO class. This includes built-in IO and IO-like objects like File, Tempfile and StringIO.

When a file is uploaded to a Rails app, in request params it will be represented by an ActionDispatch::Http::UploadedFile object, which is also an IO-like object accepted by Shrine. In other Rack applications the uploaded file will be represented as a Hash, but it can be converted into an IO-like object with the rack_file plugin.

Here are some examples of various IO-like objects that can be uploaded:

uploader.upload File.open("/path/to/file", binmode: true)   # upload from disk
uploader.upload StringIO.new("file content")                # upload from memory
uploader.upload ActionDispatch::Http::UploadedFile.new(...) # upload from Rails controller
uploader.upload Shrine.rack_file({ tempfile: tempfile })    # upload from Rack controller
uploader.upload Rack::Test::UploadedFile.new(...)           # upload from rack-test
uploader.upload Down.open("https://example.org/file")       # upload from internet
uploader.upload Shrine::UploadedFile.new(...)               # upload from Shrine storage

Uploaded file

The Shrine::UploadedFile object represents the file that was uploaded to a storage, and it's what's returned from Shrine#upload or when retrieving a record attachment. It contains the following information:

Key Description
id location of the file on the storage
storage identifier of the storage the file was uploaded to
metadata file metadata that was extracted before upload
uploaded_file = uploader.upload(file)
uploaded_file.data #=> {"id"=>"949sdjg834.jpg","storage"=>"store","metadata"=>{...}}

uploaded_file.id       #=> "949sdjg834.jpg"
uploaded_file.storage  #=> #<Shrine::Storage::S3>
uploaded_file.metadata #=> {...}

It comes with many convenient methods that delegate to the storage:

uploaded_file.url                     #=> "https://my-bucket.s3.amazonaws.com/949sdjg834.jpg"
uploaded_file.open { |io| ... }       # opens the uploaded file stream
uploaded_file.download { |file| ... } # downloads the uploaded file to disk
uploaded_file.stream(destination)     # streams uploaded content into a writable destination
uploaded_file.exists?                 #=> true
uploaded_file.delete                  # deletes the uploaded file from the storage

It also implements the IO-like interface that conforms to Shrine's IO abstraction, which allows it to be uploaded again to other storages.

uploaded_file.read   # returns content of the uploaded file
uploaded_file.eof?   # returns true if the whole IO was read
uploaded_file.rewind # rewinds the IO
uploaded_file.close  # closes the IO

For more details, see the Retrieving Uploads guide and Shrine::UploadedFile API docs.

Attachment

Storage objects, uploaders, and uploaded file objects are Shrine's foundational components. To help you actually attach uploaded files to database records in your application, Shrine comes with a high-level attachment interface built on top of these components.

There are plugins for hooking into most database libraries, and in case of ActiveRecord and Sequel the plugin will automatically tie the attached files to records' lifecycles. But you can also use Shrine just with plain old Ruby objects.

Shrine.plugin :sequel # :activerecord
class Photo < Sequel::Model # ActiveRecord::Base
  include ImageUploader::Attachment.new(:image) #
  include ImageUploader::Attachment(:image)     # these are all equivalent
  include ImageUploader[:image]                 #
end

You can choose whichever of these syntaxes you prefer. Either of these will create a Shrine::Attachment module with attachment methods for the specified attribute, which then get added to your model when you include it:

Method Description
#image= uploads the file to temporary storage and serializes the result into image_data
#image returns Shrine::UploadedFile instantiated from image_data
#image_url calls url on the attachment if it's present, otherwise returns nil
#image_attacher returns instance of Shrine::Attacher which handles the attaching

The ORM plugin that we loaded adds appropriate callbacks. For example, saving the record uploads the attachment to permanent storage, while deleting the record deletes the attachment.

# no file is attached
photo.image #=> nil

# the assigned file is cached to temporary storage and written to `image_data` column
photo.image = File.open("waterfall.jpg")
photo.image      #=> #<Shrine::UploadedFile @data={...}>
photo.image_url  #=> "/uploads/cache/0sdfllasfi842.jpg"
photo.image_data #=> '{"id":"0sdfllasfi842.jpg","storage":"cache","metadata":{...}}'

# the cached file is promoted to permanent storage and saved to `image_data` column
photo.save
photo.image      #=> #<Shrine::UploadedFile @data={...}>
photo.image_url  #=> "/uploads/store/l02kladf8jlda.jpg"
photo.image_data #=> '{"id":"l02kladf8jlda.jpg","storage":"store","metadata":{...}}'

# the attached file is deleted with the record
photo.destroy
photo.image.exists? #=> false

If there is already a file attached and a new file is attached, the previous attachment will get deleted when the record gets saved.

photo.update(image: new_file) # changes the attachment and deletes previous
photo.update(image: nil)      # removes the attachment and deletes previous

Attacher

The model attachment attributes and callbacks added by Shrine::Attachment just delegate the behaviour to their underlying Shrine::Attacher object.

photo.image_attacher #=> #<Shrine::Attacher>

The Shrine::Attacher object can be instantiated and used directly:

attacher = ImageUploader::Attacher.new(photo, :image)

attacher.assign(file) # equivalent to `photo.image = file`
attacher.get          # equivalent to `photo.image`
attacher.url          # equivalent to `photo.image_url`

The attacher is what drives attaching files to model instances; you can use it as a more explicit alternative to models' attachment interface, or simply when you need something that's not available through the attachment methods.

You can do things such as change the temporary and permanent storage the attacher uses, or upload files directly to permanent storage. See the Using Attacher guide for more details.

Plugin system

By default Shrine comes with a small core which provides only the essential functionality. All additional features are available via plugins, which also ship with Shrine. This way you can choose exactly what and how much Shrine does for you, and you load the code only for features that you use.

Shrine.plugin :logging # adds logging

Plugins add behaviour by extending Shrine core classes via module inclusion, and many of them also accept configuration options. The plugin system respects inheritance, so you can choose to load a plugin globally or per uploader.

class ImageUploader < Shrine
  plugin :store_dimensions # extract image dimensions only for this uploader and its descendants
end

If you want to extend Shrine functionality with custom behaviour, you can also create your own plugin.

Metadata

Shrine automatically extracts some basic file metadata and saves them to the Shrine::UploadedFile. You can access them through the #metadata hash or via metadata methods:

uploaded_file.metadata #=>
# {
#   "filename" => "matrix.mp4",
#   "mime_type" => "video/mp4",
#   "size" => 345993,
# }

uploaded_file.original_filename #=> "matrix.mp4"
uploaded_file.extension         #=> "mp4"
uploaded_file.mime_type         #=> "video/mp4"
uploaded_file.size              #=> 345993

MIME type

By default, mime_type metadata will be inherited from the #content_type attribute of the uploaded file, which is generally not secure and will trigger a warning. You can load the determine_mime_type plugin to have MIME type extracted from file content instead.

Shrine.plugin :determine_mime_type
photo = Photo.create(image: StringIO.new("<?php ... ?>"))
photo.image.mime_type #=> "text/x-php"

By the default the UNIX file utility is used to determine the MIME type, but you can also choose a different analyzer, see the determine_mime_type docs for more details.

Other metadata

In addition to basic metadata, you can also extract image dimensions, calculate signatures, and in general extract any custom metadata. Check out the Extracting Metadata guide for more details.

Processing

Shrine allows you to process attached files either "on upload" or "on-the-fly". For example, if your app is accepting image uploads, you can generate a pre-defined set of of thumbnails as soon as the image is attached to a record ("on upload"), or you can generate necessary thumbnails dynamically as they're needed ("on-the-fly").

For image processing it's recommended to use the ImageProcessing gem, which is a high-level wrapper for processing with ImageMagick (via MiniMagick) or libvips (via ruby-vips).

Processing on upload

For processing "on upload", you can intercept when a cached file is being uploaded to permanent storage, and perform any file processing you might want. The processing plugin provides the promotion hook, while the versions plugin enables handling a hash of versions.

$ brew install imagemagick
# Gemfile
gem "image_processing", "~> 1.2"
require "image_processing/mini_magick"

class ImageUploader < Shrine
  plugin :processing # allows hooking into promoting
  plugin :versions   # enable Shrine to handle a hash of files
  plugin :delete_raw # delete processed files after uploading

  process(:store) do |io, context|
    versions = { original: io } # retain original

    io.download do |original|
      pipeline = ImageProcessing::MiniMagick.source(original)

      versions[:large]  = pipeline.resize_to_limit!(800, 800)
      versions[:medium] = pipeline.resize_to_limit!(500, 500)
      versions[:small]  = pipeline.resize_to_limit!(300, 300)
    end

    versions # return the hash of processed files
  end
end

After the files are uploaded, their data is saved into the <attachment>_data column, and the attachment getter will read them as a Hash of Shrine::UploadedFile objects.

photo = Photo.create(image: file) # processing is triggered
photo.image #=>
# {
#   :original => #<Shrine::UploadedFile @data={"id"=>"9sd84.jpg", ...}>,
#   :large    => #<Shrine::UploadedFile @data={"id"=>"lg043.jpg", ...}>,
#   :medium   => #<Shrine::UploadedFile @data={"id"=>"kd9fk.jpg", ...}>,
#   :small    => #<Shrine::UploadedFile @data={"id"=>"932fl.jpg", ...}>,
# }

photo.image[:medium]           #=> #<Shrine::UploadedFile>
photo.image[:medium].url       #=> "/uploads/store/lg043.jpg"
photo.image[:medium].size      #=> 5825949
photo.image[:medium].mime_type #=> "image/jpeg"

photo.image_url(:large) # returns version URL with fallbacks in case version is missing

By default processing is executed synchronously, but you can choose to delay it into a background job. You can also do any other type of file processing you want, see the File Processing guide for more details.

Processing on-the-fly

On-the-fly processing is provided by the derivation_endpoint plugin. It comes with a mountable Rack app which applies processing on request and returns processed files.

To set it up, we mount the Rack app in our router on a chosen path prefix, configure the plugin with a secret key and that path prefix, and define processing we want to perform:

$ brew install imagemagick
# Gemfile
gem "image_processing", "~> 1.2"
# config/routes.rb (Rails)
Rails.application.routes.draw do
  # ...
  mount ImageUploader.derivation_endpoint => "/derivations/image"
end
require "image_processing/mini_magick"

class ImageUploader < Shrine
  plugin :derivation_endpoint,
    secret_key: "<YOUR SECRET KEY>",
    prefix:     "derivations/image" # needs to match the mount point in routes

  derivation :thumbnail do |file, width, height|
    ImageProcessing::MiniMagick
      .source(file)
      .resize_to_limit!(width.to_i, height.to_i)
  end
end

Now we can generate URLs from attached files that will perform the desired processing:

photo.image.derivation_url(:thumbnail, "600", "400")
#=> "/derivations/image/thumbnail/600/400/eyJpZCI6ImZvbyIsInN0b3JhZ2UiOiJzdG9yZSJ9?signature=..."

The on-the-fly processing feature is highly customizable, see the derivation_endpoint plugin documentation for more details.

Validation

Shrine can perform file validations for files assigned to the model, with validation_helpers plugin providing some common validation methods:

class DocumentUploader < Shrine
  plugin :validation_helpers

  Attacher.validate do
    validate_max_size 5*1024*1024, message: "is too large (max is 5 MB)"
    validate_mime_type_inclusion %w[application/pdf]
  end
end
user = User.new
user.cv = File.open("cv.pdf", "rb")
user.valid? #=> false
user.errors.to_hash #=> {:cv=>["is too large (max is 5 MB)"]}

For more details, see the File Validation guide and validation_helpers plugin docs.

Location

Shrine automatically generated random locations before uploading files. By default the hierarchy is flat, meaning all files are stored in the root directory of the storage. The pretty_location plugin provides a good default hierarchy, but you can also override #generate_location with a custom implementation:

class ImageUploader < Shrine
  def generate_location(io, context)
    type  = context[:record].class.name.downcase if context[:record]
    style = context[:version] == :original ? "originals" : "thumbs" if context[:version]
    name  = super # the default unique identifier

    [type, style, name].compact.join("/")
  end
end
uploads/
  photos/
    originals/
      la98lda74j3g.jpg
    thumbs/
      95kd8kafg80a.jpg
      ka8agiaf9gk4.jpg

Note that there should always be a random component in the location, so that the ORM dirty tracking is detected properly. Inside #generate_location you can also access the extracted metadata through context[:metadata].

Direct uploads

To improve the user experience, it's recommended to upload files asynchronously as soon as the user selects them. The direct uploads would go to temporary storage, just like in the synchronous flow. Then, instead of attaching a raw file to your model, you assign the cached file JSON data.

# in the regular synchronous flow
photo.image = file

# in the direct upload flow
photo.image = '{"id":"...","storage":"cache","metadata":{...}}'

On the client side it's highly recommended to use Uppy 🐶, a very flexible modern JavaScript file upload library that happens to integrate nicely with Shrine.

Simple direct upload

The simplest approach is to upload directly to an endpoint in your app, which forwards uploads to the specified storage. The upload_endpoint Shrine plugin provides a mountable Rack app that implements this endpoint:

Shrine.plugin :upload_endpoint
# config/routes.rb (Rails)
Rails.application.routes.draw do
  # ...
  mount ImageUploader.upload_endpoint(:cache) => "/images/upload" # POST /images/upload
end

Then you can configure Uppy's XHR Upload plugin to upload to this endpoint. See this walkthrough for adding simple direct uploads from scratch, it includes a complete JavaScript example (there is also the Roda / Rails demo app).

Presigned direct upload

For better performance, you can also upload files directly to your cloud storage service (AWS S3, Google Cloud Storage etc). For this, your temporary storage needs to be your cloud service:

require "shrine/storage/s3"

Shrine.storages = {
  cache: Shrine::Storage::S3.new(prefix: "cache", **s3_options),
  store: Shrine::Storage::S3.new(**s3_options)
}

In this flow, the client needs to first fetch upload parameters from the server, and then use these parameters for the upload to the cloud service. The presign_endpoint Shrine plugin provides a mountable Rack app that generates upload parameters:

Shrine.plugin :presign_endpoint
# config/routes.rb (Rails)
Rails.application.routes.draw do
  # ...
  mount Shrine.presign_endpoint(:cache) => "/s3/params" # GET /s3/params
end

Then you can configure Uppy's AWS S3 plugin to fetch params from your endpoint before uploading to S3. See this walkthrough for adding direct uploads to S3 from scratch, it includes a complete JavaScript example (there is also the Roda / Rails demo). See also the Direct Uploads to S3 guide for more details.

Resumable direct upload

If your app is accepting large uploads, you can improve resilience by making the uploads resumable. This can significantly improve experience for users on slow and flaky internet connections.

Uppy S3 Multipart

You can achieve resumable uploads directly to S3 with the AWS S3 Multipart Uppy plugin, accompanied with uppy_s3_multipart Shrine plugin provided by the uppy-s3_multipart gem.

# Gemfile
gem "uppy-s3_multipart", "~> 0.3"
Shrine.plugin :uppy_s3_multipart
# config/routes.rb (Rails)
Rails.application.routes.draw do
  # ...
  mount Shrine.uppy_s3_multipart(:cache) => "/s3/multipart"
end

See the uppy-s3_multipart docs for more details.

Tus protocol

If you want a more generic approach, you can build your resumable uploads on tus – an open resumable upload protocol. On the server side you can use the tus-ruby-server gem, on the client side Uppy's Tus plugin, and the shrine-tus gem for the glue.

# Gemfile
gem "tus-server", "~> 2.0"
gem "shrine-tus", "~> 1.2"
require "shrine/storage/tus"

Shrine.storages = {
  cache: Shrine::Storage::Tus.new, # tus server acts as temporary storage
  store: ...,                      # your permanent storage
}
# config/routes.rb (Rails)
Rails.application.routes.draw do
  # ...
  mount Tus::Server => "/files"
end

See this walkthrough for adding tus-powered resumable uploads from scratch, it includes a complete JavaScript example (there is also a demo app). See also shrine-tus and tus-ruby-server docs for more details.

Backgrounding

Shrine allows you to put file deletion and promotion of cached files to permanent storage into a background job. The backgrounding plugin provides hooks for plugging in your favourite backgrounding library.

Shrine.plugin :backgrounding
Shrine::Attacher.promote { |data| PromoteJob.perform_async(data) }
Shrine::Attacher.delete { |data| DeleteJob.perform_async(data) }
class PromoteJob
  include Sidekiq::Worker
  def perform(data)
    Shrine::Attacher.promote(data)
  end
end
class DeleteJob
  include Sidekiq::Worker
  def perform(data)
    Shrine::Attacher.delete(data)
  end
end

Clearing cache

Shrine doesn't automatically delete files uploaded to temporary storage, instead you should set up a separate recurring task that will automatically delete old cached files.

Most Shrine storage classes come with a #clear! method, which you can call in a recurring script. For FileSystem and S3 storage it would look like this:

# FileSystem storage
file_system = Shrine.storages[:cache]
file_system.clear! { |path| path.mtime < Time.now - 7*24*60*60 } # delete files older than 1 week
# S3 storage
s3 = Shrine.storages[:cache]
s3.clear! { |object| object.last_modified < Time.now - 7*24*60*60 } # delete files older than 1 week

Inspiration

Shrine was heavily inspired by Refile and Roda. From Refile it borrows the idea of "backends" (here named "storages"), attachment interface, and direct uploads. From Roda it borrows the implementation of an extensible plugin system.

Similar libraries

  • Paperclip
  • CarrierWave
  • Dragonfly
  • Refile
  • Active Storage

Code of Conduct

Everyone interacting in the Shrine project’s codebases, issue trackers, and mailing lists is expected to follow the Shrine code of conduct.

License

The gem is available as open source under the terms of the MIT License.