Class: Dis::Layer

Inherits:
Object
  • Object
show all
Includes:
Logging
Defined in:
lib/dis/layer.rb

Overview

Dis Layer

Represents a layer of storage. Wraps a Fog::Storage connection; any provider supported by Fog should be usable.

Examples:

Local storage layer

Dis::Layer.new(
  Fog::Storage.new(
    provider: "Local",
    local_root: Rails.root.join("db/dis")
  ),
  path: Rails.env
)

Delayed layer on Amazon S3

Dis::Layer.new(
  Fog::Storage.new(
    provider: "AWS",
    aws_access_key_id: YOUR_AWS_ACCESS_KEY_ID,
    aws_secret_access_key: YOUR_AWS_SECRET_ACCESS_KEY
  ),
  path: "my_bucket",
  delayed: true
)

Instance Attribute Summary collapse

Instance Method Summary collapse

Methods included from Logging

#debug_log

Constructor Details

#initialize(connection, options = {}) ⇒ Layer

Returns a new instance of Layer.

Parameters:

  • connection (Fog::Storage) —

    a Fog storage connection

  • options (Hash) (defaults to: {}) —

    layer configuration options

Options Hash (options):

  • :delayed (Boolean) — default: false —

    process writes asynchronously via ActiveJob

  • :readonly (Boolean) — default: false —

    only allow reads

  • :public (Boolean) — default: false —

    set the public readable flag on stored objects (provider-dependent)

  • :path (String) — default: nil —

    directory or bucket name

  • :cache (Integer, false) — default: false —

    enable bounded cache with this soft size limit in bytes. Cannot be combined with :delayed or :readonly

Raises:

  • (ArgumentError) —

    if :cache is combined with :delayed or :readonly



47
48
49
50
51
52
53
54
55
56
 
# File 'lib/dis/layer.rb', line 47

def initialize(connection, options = {})
  options     = default_options.merge(options)
  @connection = connection
  @delayed    = options[:delayed]
  @readonly   = options[:readonly]
  @public     = options[:public]
  @path       = options[:path]
  @cache      = options[:cache]
  validate_cache_options!
end

Instance Attribute Details

#connection ⇒ Fog::Storage (readonly)

Returns the underlying Fog connection.

Returns:

  • (Fog::Storage) —

    the underlying Fog connection



32
33
34
 
# File 'lib/dis/layer.rb', line 32

def connection
  @connection
end

Instance Method Details

#cache? ⇒ Boolean

Returns true if the layer is a cache layer.

Returns:

  • (Boolean) 


96
97
98
 
# File 'lib/dis/layer.rb', line 96

def cache?
  !!@cache
end

#cached_files ⇒ Array<Hash>

Returns cached file entries sorted by mtime ascending (oldest first).

Returns:

  • (Array<Hash>) —

    each entry has keys :path (Pathname), :type (String), :key (String), :mtime (Time), :size (Integer)



126
127
128
129
130
131
132
133
134
135
 
# File 'lib/dis/layer.rb', line 126

def cached_files
  return [] unless connection.respond_to?(:local_root)

  root = local_root_path
  return [] unless root.exist?

  entries = root.glob("**/*").select(&:file?)
  entries.filter_map { |f| cached_file_entry(f, root) }
         .sort_by { |e| e[:mtime] }
end

#delayed? ⇒ Boolean

Returns true if the layer is a delayed layer.

Returns:

  • (Boolean) 


61
62
63
 
# File 'lib/dis/layer.rb', line 61

def delayed?
  @delayed
end

#delete(type, key) ⇒ Boolean

Deletes a file from the store.

Parameters:

  • type (String) —

    the type scope

  • key (String) —

    the content hash

Returns:

  • (Boolean) —

    true if the file was deleted, false if not found

Raises:



244
245
246
247
248
249
250
 
# File 'lib/dis/layer.rb', line 244

def delete(type, key)
  raise Dis::Errors::ReadOnlyError if readonly?

  debug_log("Delete #{type}/#{key} from #{name}") do
    delete!(type, key)
  end
end

#existing(type, keys) ⇒ Array<String>

Returns all the given keys that exist in the layer.

Parameters:

  • type (String) —

    the type scope

  • keys (Array<String>) —

    content hashes to check

Returns:

  • (Array<String>) —

    the subset of keys that exist



163
164
165
166
167
168
169
170
 
# File 'lib/dis/layer.rb', line 163

def existing(type, keys)
  return [] if keys.empty?

  futures = keys.map do |key|
    Concurrent::Promises.future { key if exists?(type, key) }
  end
  futures.filter_map(&:value!)
end

#exists?(type, key) ⇒ Boolean

Returns true if an object with the given key exists.

Parameters:

  • type (String) —

    the type scope

  • key (String) —

    the content hash

Returns:

  • (Boolean) 


196
197
198
199
200
201
202
 
# File 'lib/dis/layer.rb', line 196

def exists?(type, key)
  if directory(type, key)&.files&.head(key_component(type, key))
    true
  else
    false
  end
end

#file_path(type, key) ⇒ String?

Returns the absolute file path for a locally stored file, or nil if the provider is not local or the file does not exist.

Parameters:

  • type (String) —

    the type scope

  • key (String) —

    the content hash

Returns:

  • (String, nil) 


226
227
228
229
230
231
232
233
234
235
 
# File 'lib/dis/layer.rb', line 226

def file_path(type, key)
  return unless connection.respond_to?(:local_root)
  return unless exists?(type, key)

  File.join(
    connection.local_root,
    directory_component(type, key),
    key_component(type, key)
  )
end

#get(type, key) ⇒ Fog::Model?

Retrieves a file from the store.

Parameters:

  • type (String) —

    the type scope

  • key (String) —

    the content hash

Returns:

  • (Fog::Model, nil) —

    the file, or nil if not found



209
210
211
212
213
214
215
216
217
218
 
# File 'lib/dis/layer.rb', line 209

def get(type, key)
  dir = directory(type, key)
  return unless dir

  result = debug_log("Get #{type}/#{key} from #{name}") do
    dir.files.get(key_component(type, key))
  end
  touch_file(type, key) if result && cache?
  result
end

#immediate? ⇒ Boolean

Returns true if the layer isn’t a delayed layer.

Returns:

  • (Boolean) 


68
69
70
 
# File 'lib/dis/layer.rb', line 68

def immediate?
  !delayed?
end

#max_size ⇒ Integer?

Returns the cache size limit in bytes, or nil if not a cache.

Returns:

  • (Integer, nil) 


103
104
105
 
# File 'lib/dis/layer.rb', line 103

def max_size
  @cache if cache?
end

#name ⇒ String

Returns a name for the layer.

Examples:

layer.name # => "Fog::Storage::Local::Real/development"

Returns:

  • (String) 


258
259
260
 
# File 'lib/dis/layer.rb', line 258

def name
  "#{connection.class.name}/#{path}"
end

#public? ⇒ Boolean

Returns true if the layer is public.

Returns:

  • (Boolean) 


75
76
77
 
# File 'lib/dis/layer.rb', line 75

def public?
  @public
end

#readonly? ⇒ Boolean

Returns true if the layer is read only.

Returns:

  • (Boolean) 


82
83
84
 
# File 'lib/dis/layer.rb', line 82

def readonly?
  @readonly
end

#size ⇒ Integer

Returns the total size in bytes of all files stored locally. Returns 0 for non-local providers.

Returns:

  • (Integer) 


111
112
113
114
115
116
117
118
 
# File 'lib/dis/layer.rb', line 111

def size
  return 0 unless connection.respond_to?(:local_root)

  root = local_root_path
  return 0 unless root.exist?

  root.glob("**/*").sum { |f| f.file? ? f.size : 0 }
end

#store(type, key, file) ⇒ Fog::Model

Stores a file. The key must be a hex digest of the file content. If an object with the supplied hash already exists, no action will be performed.

Examples:

key = Digest::SHA1.file(file.path).hexdigest
layer.store("documents", key, file)

Parameters:

  • type (String) —

    the type scope

  • key (String) —

    the content hash

  • file (File, IO, String, Fog::Model) —

    the content

Returns:

  • (Fog::Model) —

    the stored file

Raises:



150
151
152
153
154
155
156
 
# File 'lib/dis/layer.rb', line 150

def store(type, key, file)
  raise Dis::Errors::ReadOnlyError if readonly?

  debug_log("Store #{type}/#{key} to #{name}") do
    store!(type, key, file)
  end
end

#stored_keys(type) ⇒ Array<String>

Returns all content hashes stored under the given type.

Parameters:

  • type (String) —

    the type scope

Returns:

  • (Array<String>) —

    content hashes



176
177
178
179
180
181
182
183
184
185
186
187
188
189
 
# File 'lib/dis/layer.rb', line 176

def stored_keys(type)
  dir = connection.directories.get(path || "")
  return [] unless dir

  prefix = "#{type}/"
  dir.files.filter_map do |file|
    next unless file.key.start_with?(prefix)

    parts = file.key.delete_prefix(prefix).split("/")
    next unless parts.length == 2

    "#{parts[0]}#{parts[1]}"
  end
end

#writeable? ⇒ Boolean

Returns true if the layer is writeable.

Returns:

  • (Boolean) 


89
90
91
 
# File 'lib/dis/layer.rb', line 89

def writeable?
  !readonly?
end