Class: Shrine::Storage::S3

Inherits:
Object
  • Object
show all
Includes:
ClientSideEncryption
Defined in:
lib/shrine/storage/s3.rb

Defined Under Namespace

Modules: ClientSideEncryption

Constant Summary collapse

MAX_MULTIPART_PARTS =
10_000
MIN_PART_SIZE =
5*1024*1024
MULTIPART_THRESHOLD =
{ upload: 15*1024*1024, copy: 100*1024*1024 }

Instance Attribute Summary collapse

Attributes included from ClientSideEncryption

#encryption_client

Instance Method Summary collapse

Constructor Details

#initialize(bucket:, client: nil, prefix: nil, upload_options: {}, multipart_threshold: {}, signer: nil, public: nil, **s3_options) ⇒ S3

Initializes a storage for uploading to S3. All options are forwarded to [`Aws::S3::Client#initialize`], except the following:

:bucket : (Required). Name of the S3 bucket.

:client : By default an `Aws::S3::Client` instance is created internally from

additional options, but you can use this option to provide your own
client. This can be an `Aws::S3::Client` or an
`Aws::S3::Encryption::Client` object.

:prefix : “Directory” inside the bucket to store files into.

:upload_options : Additional options that will be used for uploading files, they will

be passed to [`Aws::S3::Object#put`], [`Aws::S3::Object#copy_from`]
and [`Aws::S3::Bucket#presigned_post`].

:multipart_threshold : If the input file is larger than the specified size, a parallelized

multipart will be used for the upload/copy. Defaults to
`{upload: 15*1024*1024, copy: 100*1024*1024}` (15MB for upload
requests, 100MB for copy requests).

In addition to specifying the `:bucket`, you'll also need to provide AWS credentials. The most common way is to provide them directly via `:access_key_id`, `:secret_access_key`, and `:region` options. But you can also use any other way of authentication specified in the [AWS SDK documentation][configuring AWS SDK].

[`Aws::S3::Object#put`]: docs.aws.amazon.com/sdk-for-ruby/v3/api/Aws/S3/Object.html#put-instance_method [`Aws::S3::Object#copy_from`]: docs.aws.amazon.com/sdk-for-ruby/v3/api/Aws/S3/Object.html#copy_from-instance_method [`Aws::S3::Bucket#presigned_post`]: docs.aws.amazon.com/sdk-for-ruby/v3/api/Aws/S3/Object.html#presigned_post-instance_method [`Aws::S3::Client#initialize`]: docs.aws.amazon.com/sdk-for-ruby/v3/api/Aws/S3/Client.html#initialize-instance_method [configuring AWS SDK]: docs.aws.amazon.com/sdk-for-ruby/v3/developer-guide/setup-config.html

Raises:

  • (ArgumentError)

61
62
63
64
65
66
67
68
69
70
71
# File 'lib/shrine/storage/s3.rb', line 61

def initialize(bucket:, client: nil, prefix: nil, upload_options: {}, multipart_threshold: {}, signer: nil, public: nil, **s3_options)
  raise ArgumentError, "the :bucket option is nil" unless bucket

  @client = client || Aws::S3::Client.new(**s3_options)
  @bucket = Aws::S3::Bucket.new(name: bucket, client: @client)
  @prefix = prefix
  @upload_options = upload_options
  @multipart_threshold = MULTIPART_THRESHOLD.merge(multipart_threshold)
  @signer = signer
  @public = public
end

Instance Attribute Details

#bucketObject (readonly)

Returns the value of attribute bucket.


17
18
19
# File 'lib/shrine/storage/s3.rb', line 17

def bucket
  @bucket
end

#clientObject (readonly)

Returns the value of attribute client.


17
18
19
# File 'lib/shrine/storage/s3.rb', line 17

def client
  @client
end

#prefixObject (readonly)

Returns the value of attribute prefix.


17
18
19
# File 'lib/shrine/storage/s3.rb', line 17

def prefix
  @prefix
end

#publicObject (readonly)

Returns the value of attribute public.


17
18
19
# File 'lib/shrine/storage/s3.rb', line 17

def public
  @public
end

#signerObject (readonly)

Returns the value of attribute signer.


17
18
19
# File 'lib/shrine/storage/s3.rb', line 17

def signer
  @signer
end

#upload_optionsObject (readonly)

Returns the value of attribute upload_options.


17
18
19
# File 'lib/shrine/storage/s3.rb', line 17

def upload_options
  @upload_options
end

Instance Method Details

#clear!(&block) ⇒ Object

If block is given, deletes all objects from the storage for which the block evaluates to true. Otherwise deletes all objects from the storage.

s3.clear!
# or
s3.clear! { |object| object.last_modified < Time.now - 7*24*60*60 }

207
208
209
210
211
212
# File 'lib/shrine/storage/s3.rb', line 207

def clear!(&block)
  objects_to_delete = bucket.objects(prefix: prefix)
  objects_to_delete = objects_to_delete.lazy.select(&block) if block

  delete_objects(objects_to_delete)
end

#delete(id) ⇒ Object

Deletes the file from the storage.


187
188
189
# File 'lib/shrine/storage/s3.rb', line 187

def delete(id)
  object(id).delete
end

#delete_prefixed(delete_prefix) ⇒ Object

Deletes objects at keys starting with the specified prefix.

s3.delete_prefixed("somekey/derivatives/")

194
195
196
197
198
199
# File 'lib/shrine/storage/s3.rb', line 194

def delete_prefixed(delete_prefix)
  # We need to make sure to combine with storage prefix, and
  # that it ends in '/' cause S3 can be squirrely about matching interior.
  delete_prefix = delete_prefix.chomp("/") + "/"
  bucket.objects(prefix: [*prefix, delete_prefix].join("/")).batch_delete!
end

#exists?(id) ⇒ Boolean

Returns true file exists on S3.

Returns:

  • (Boolean)

115
116
117
# File 'lib/shrine/storage/s3.rb', line 115

def exists?(id)
  object(id).exists?
end

#object(id) ⇒ Object

Returns an `Aws::S3::Object` for the given id.


215
216
217
# File 'lib/shrine/storage/s3.rb', line 215

def object(id)
  bucket.object(object_key(id))
end

#open(id, rewindable: true, **options) ⇒ Object

Returns a `Down::ChunkedIO` object that downloads S3 object content on-demand. By default, read content will be cached onto disk so that it can be rewinded, but if you don't need that you can pass `rewindable: false`.

Any additional options are forwarded to [`Aws::S3::Object#get`].

[`Aws::S3::Object#get`]: docs.aws.amazon.com/sdk-for-ruby/v3/api/Aws/S3/Object.html#get-instance_method


106
107
108
109
110
111
112
# File 'lib/shrine/storage/s3.rb', line 106

def open(id, rewindable: true, **options)
  chunks, length = get(id, **options)

  Down::ChunkedIO.new(chunks: chunks, rewindable: rewindable, size: length)
rescue Aws::S3::Errors::NoSuchKey
  raise Shrine::FileNotFound, "file #{id.inspect} not found on storage"
end

#presign(id, method: :post, **presign_options) ⇒ Object

Returns URL, params, headers, and verb for direct uploads.

s3.presign("key") #=>
# {
#   url: "https://my-bucket.s3.amazonaws.com/...",
#   fields: { ... },  # blank for PUT presigns
#   headers: { ... }, # blank for POST presigns
#   method: "post",
# }

By default it calls [`Aws::S3::Object#presigned_post`] which generates data for a POST request, but you can also specify `method: :put` for PUT uploads which calls [`Aws::S3::Object#presigned_url`].

s3.presign("key", method: :post) # for POST upload (default)
s3.presign("key", method: :put)  # for PUT upload

Any additional options are forwarded to the underlying AWS SDK method.

[`Aws::S3::Object#presigned_post`]: docs.aws.amazon.com/sdk-for-ruby/v3/api/Aws/S3/Object.html#presigned_post-instance_method [`Aws::S3::Object#presigned_url`]: docs.aws.amazon.com/sdk-for-ruby/v3/api/Aws/S3/Object.html#presigned_url-instance_method


176
177
178
179
180
181
182
183
184
# File 'lib/shrine/storage/s3.rb', line 176

def presign(id, method: :post, **presign_options)
  options = {}
  options[:acl] = "public-read" if public

  options.merge!(@upload_options)
  options.merge!(presign_options)

  send(:"presign_#{method}", id, options)
end

#upload(io, id, shrine_metadata: {}, **upload_options) ⇒ Object

If the file is an UploadedFile from S3, issues a COPY command, otherwise uploads the file. For files larger than `:multipart_threshold` a multipart upload/copy will be used for better performance and more resilient uploads.

It assigns the correct “Content-Type” taken from the MIME type, because by default S3 sets everything to “application/octet-stream”.


80
81
82
83
84
85
86
87
88
89
90
91
92
93
94
95
96
# File 'lib/shrine/storage/s3.rb', line 80

def upload(io, id, shrine_metadata: {}, **upload_options)
  content_type, filename = .values_at("mime_type", "filename")

  options = {}
  options[:content_type] = content_type if content_type
  options[:content_disposition] = ContentDisposition.inline(filename) if filename
  options[:acl] = "public-read" if public

  options.merge!(@upload_options)
  options.merge!(upload_options)

  if copyable?(io)
    copy(io, id, **options)
  else
    put(io, id, **options)
  end
end

#url(id, public: self.public, host: nil, **options) ⇒ Object

Returns the presigned URL to the file.

:host : This option replaces the host part of the returned URL, and is

typically useful for setting CDN hosts (e.g.
`http://abc123.cloudfront.net`)

:public : Returns the unsigned URL to the S3 object. This requires the S3

object to be public.

All other options are forwarded to [`Aws::S3::Object#presigned_url`] or [`Aws::S3::Object#public_url`].

[`Aws::S3::Object#presigned_url`]: docs.aws.amazon.com/sdk-for-ruby/v3/api/Aws/S3/Object.html#presigned_url-instance_method [`Aws::S3::Object#public_url`]: docs.aws.amazon.com/sdk-for-ruby/v3/api/Aws/S3/Object.html#public_url-instance_method


135
136
137
138
139
140
141
142
143
144
145
146
147
148
149
150
151
152
153
# File 'lib/shrine/storage/s3.rb', line 135

def url(id, public: self.public, host: nil, **options)
  if public || signer
    url = object(id).public_url(**options)
  else
    url = object(id).presigned_url(:get, **options)
  end

  if host
    uri = URI.parse(url)
    uri.path = uri.path.match(/^\/#{bucket.name}/).post_match unless uri.host.include?(bucket.name)
    url = URI.join(host, uri.request_uri[1..-1]).to_s
  end

  if signer
    url = signer.call(url, **options)
  end

  url
end