Class: COS::Bucket

Inherits:
Object
  • Object
show all
Includes:
Logging
Defined in:
lib/cos/bucket.rb

Constant Summary collapse

MIN_UPLOAD_SLICE_SIZE =

最小完整上传大小

10 * 1024 * 1024
MIN_DOWNLOAD_SLICE_SIZE =

最小下载分块大小

5 * 1024 * 1024
DEFAULT_UPLOAD_RETRY =

默认上传重试次数

10
DEFAULT_DOWNLOAD_RETRY =

默认下载重试次数

10

Constants included from Logging

Logging::DEFAULT_LOG_FILE, Logging::MAX_NUM_LOG, Logging::ROTATE_SIZE

Instance Attribute Summary collapse

Instance Method Summary collapse

Methods included from Logging

#logger, set_logger

Constructor Details

#initialize(client, bucket_name = nil) ⇒ COS::Bucket

Note:

SDK会自动获取bucket的信息,包括读取权限等并进行缓存 如需在后台修改了bucket信息请重新初始化Client

初始化

Raises:


40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
# File 'lib/cos/bucket.rb', line 40

def initialize(client, bucket_name = nil)
  @client      = client
  @bucket_name = client.config.get_bucket(bucket_name)

  # 使用stat API 获取根目录信息可获取到bucket信息
  data = client.api.stat('/', bucket: bucket_name)
  @authority             = data[:authority]
  @bucket_type           = data[:bucket_type]
  @need_preview          = data[:need_preview]
  @refers                = data[:refers]
  @migrate_source_domain = data[:migrate_source_domain]
  @blackrefers           = data[:blackrefers]
  @brower_exec           = data[:brower_exec]
  @cnames                = data[:cnames]
  @nugc_flag             = data[:nugc_flag]
end

Instance Attribute Details

#authorityObject (readonly)

Returns the value of attribute authority


11
12
13
# File 'lib/cos/bucket.rb', line 11

def authority
  @authority
end

#blackrefersObject (readonly)

Returns the value of attribute blackrefers


11
12
13
# File 'lib/cos/bucket.rb', line 11

def blackrefers
  @blackrefers
end

#brower_execObject (readonly)

Returns the value of attribute brower_exec


11
12
13
# File 'lib/cos/bucket.rb', line 11

def brower_exec
  @brower_exec
end

#bucket_nameObject (readonly)

Returns the value of attribute bucket_name


11
12
13
# File 'lib/cos/bucket.rb', line 11

def bucket_name
  @bucket_name
end

#bucket_typeObject (readonly)

Returns the value of attribute bucket_type


11
12
13
# File 'lib/cos/bucket.rb', line 11

def bucket_type
  @bucket_type
end

#clientObject (readonly)

Returns the value of attribute client


11
12
13
# File 'lib/cos/bucket.rb', line 11

def client
  @client
end

#cnamesObject (readonly)

Returns the value of attribute cnames


11
12
13
# File 'lib/cos/bucket.rb', line 11

def cnames
  @cnames
end

#migrate_source_domainObject (readonly)

Returns the value of attribute migrate_source_domain


11
12
13
# File 'lib/cos/bucket.rb', line 11

def migrate_source_domain
  @migrate_source_domain
end

#need_previewObject (readonly)

Returns the value of attribute need_preview


11
12
13
# File 'lib/cos/bucket.rb', line 11

def need_preview
  @need_preview
end

#nugc_flagObject (readonly)

Returns the value of attribute nugc_flag


11
12
13
# File 'lib/cos/bucket.rb', line 11

def nugc_flag
  @nugc_flag
end

#refersObject (readonly)

Returns the value of attribute refers


11
12
13
# File 'lib/cos/bucket.rb', line 11

def refers
  @refers
end

Instance Method Details

#complete?(path) ⇒ Boolean

判断文件是否上传完成

Examples:

puts bucket.complete?('path1/file1')

Raises:


421
422
423
# File 'lib/cos/bucket.rb', line 421

def complete?(path)
  get_file(path).complete?
end

#count(path = '') ⇒ Integer Also known as: size

获取文件及目录总数

Raises:


113
114
115
116
# File 'lib/cos/bucket.rb', line 113

def count(path = '')
  lc = list_count(path)
  lc[:total]
end

#count_dirs(path = '') ⇒ Integer

获取目录数

Raises:


139
140
141
142
# File 'lib/cos/bucket.rb', line 139

def count_dirs(path = '')
  lc = list_count(path)
  lc[:dirs]
end

#count_files(path = '') ⇒ Integer

获取文件数

Raises:


127
128
129
130
# File 'lib/cos/bucket.rb', line 127

def count_files(path = '')
  lc = list_count(path)
  lc[:files]
end

#create_folder(path, options = {}) ⇒ COS::COSDir Also known as: mkdir

创建目录

Options Hash (options):

  • :biz_attr (String)

    目录属性, 业务端维护

Raises:

See Also:


68
69
70
71
72
73
74
75
76
77
78
79
80
# File 'lib/cos/bucket.rb', line 68

def create_folder(path, options = {})
  data = client.api.create_folder(path, options.merge({bucket: bucket_name}))
  dir  = {
      mtime:    data[:mtime],
      ctime:    data[:ctime],
      name:     data[:name],
      biz_attr: options[:biz_attr],
      bucket:   self,
      path:     path
  }

  COSDir.new(dir)
end

#delete(path) ⇒ Object

Note:

非空目录及根目录不可删除,会抛出异常

删除文件或目录

Examples:

bucket.delete('path1/file')

Raises:


355
356
357
# File 'lib/cos/bucket.rb', line 355

def delete(path)
  client.api.delete(path, bucket: bucket_name)
end

#delete!(path) ⇒ Object

Note:

非空目录及根目录不可删除, 返回false

删除文件或目录, 不会抛出异常而是返回布尔值

Examples:

puts bucket.delete!('path1/file')

367
368
369
370
371
372
# File 'lib/cos/bucket.rb', line 367

def delete!(path)
  delete(path)
  true
rescue
  false
end

#download(path_or_file, file_store, options = {}) {|Float| ... } ⇒ String

下载文件, 支持断点续传, 支持多线程

Examples:

file = bucket.download('path/file1', '~/test/file1') do |p|
  puts "下载进度: #{(p*100).round(2)}%")
end
puts file

Options Hash (options):

  • :min_slice_size (Integer)

    完整下载最小文件大小, 超过此大小将会使用分片多线程断点续传

  • :download_retry (Integer)

    下载重试次数, 默认10

  • :disable_cpt (Boolean)

    是否禁用checkpoint功能,如 果设置为true,则在下载的过程中不会写checkpoint文件,这意味着 下载失败后不能断点续传,而只能重新下载整个文件。如果这个值为 true,则:cpt_file会被忽略。

  • :threads (Integer)

    多线程下载线程数, 默认为10

  • :slice_size (Integer)

    设置分片下载时每个分片的大小 默认为5 MB。

  • :cpt_file (String)

    断点续传的checkpoint文件,如果 指定的cpt文件不存在,则会在file所在目录创建一个默认的cpt文件, 命名方式为:file.cpt,其中file是用户要下载的文件名。在下载的过 程中会不断更新此文件,成功完成下载后会删除此文件;如果指定的 cpt文件已存在,则从cpt文件中记录的点继续下载。

Yields:

  • (Float)

    下载进度百分比回调, 进度值是一个0-1之间的小数

Raises:


502
503
504
505
506
507
508
509
510
511
512
513
514
515
516
517
518
519
520
521
522
523
524
525
526
527
528
529
530
531
532
533
534
535
536
537
538
539
540
# File 'lib/cos/bucket.rb', line 502

def download(path_or_file, file_store, options = {}, &block)
  min_size    = options[:min_slice_size] || MIN_DOWNLOAD_SLICE_SIZE
  retry_times = options[:download_retry] || DEFAULT_DOWNLOAD_RETRY

  # 如果传入的是一个路径需要先获取文件信息
  file = get_file(path_or_file)

  # 检查文件是否上传完整才能下载
  unless file.complete?
    raise FileUploadNotComplete, 'file upload not complete'
  end

  # 检查本地文件sha1是否一致, 如一致就已下载完成了
  if file.sha1_match?(file_store)
    logger.info("File #{file_store} exist and sha1 match, skip download.")
    return file_store
  end

  retry_loop(retry_times) do
    if file.filesize > min_size
      # 分块下载
      Download.new(
          bucket:     self,
          cos_file:   file,
          file_store: file_store,
          options:    options,
          progress:   block
      ).download

    else
      # 直接下载
      client.api.download(file.access_url, file_store, bucket: bucket_name)

    end
  end

  # 返回本地文件路径
  file_store
end

#download_all(path_or_dir, file_store_path, options = {}) {|Float| ... } ⇒ Array<String>

Note:

sdk会自动创建本地目录

批量下载目录下的全部文件(不包含子目录)

Examples:

files = bucket.download_all('path/', '~/test/') do |p|
  puts "下载进度: #{(p*100).round(2)}%")
end
puts files

Options Hash (options):

  • :min_slice_size (Integer)

    完整下载最小文件大小, 超过此大小将会使用分片多线程断点续传

  • :disable_mkdir (Integer)

    禁止自动创建本地文件夹, 默认会创建

  • :download_retry (Integer)

    下载重试次数, 默认10

  • :disable_cpt (Boolean)

    是否禁用checkpoint功能,如 果设置为true,则在下载的过程中不会写checkpoint文件,这意味着 下载失败后不能断点续传,而只能重新下载整个文件。如果这个值为 true,则:cpt_file会被忽略。

  • :threads (Integer)

    多线程下载线程数, 默认为10

  • :slice_size (Integer)

    设置分片下载时每个分片的大小 默认为5 MB。

  • :cpt_file (String)

    断点续传的checkpoint文件,如果 指定的cpt文件不存在,则会在file所在目录创建一个默认的cpt文件, 命名方式为:file.cpt,其中file是用户要下载的文件名。在下载的过 程中会不断更新此文件,成功完成下载后会删除此文件;如果指定的 cpt文件已存在,则从cpt文件中记录的点继续下载。

Yields:

  • (Float)

    下载进度百分比回调, 进度值是一个0-1之间的小数

Raises:


578
579
580
581
582
583
584
585
586
587
588
589
590
591
592
593
# File 'lib/cos/bucket.rb', line 578

def download_all(path_or_dir, file_store_path, options = {}, &block)
  local_path = Util.get_local_path(file_store_path, options[:disable_mkdir])
  dir        = get_dir(path_or_dir)
  downloaded = []

  # 遍历目录下的所有文件
  dir.list(pattern: :file_only).each do |file|
    logger.info("Begin to download file >> #{file.name}")

    downloaded << file.download("#{local_path}/#{file.name}", options, &block)

    logger.info("#{file.name} download finished")
  end

  downloaded
end

#empty?(path = '') ⇒ Boolean

目录是否是空的

Examples:

puts bucket.empty?('path1/')

Raises:


384
385
386
# File 'lib/cos/bucket.rb', line 384

def empty?(path = '')
  count(path) == 0
end

#exist?(path) ⇒ Boolean Also known as: exists?

文件或目录是否存在

Examples:

puts bucket.exist?('path1/file1')

Raises:


398
399
400
401
402
403
404
405
406
407
# File 'lib/cos/bucket.rb', line 398

def exist?(path)
  begin
    stat(path)
  rescue ServerError => e
    return false if e.error_code == -166
    raise e
  end

  true
end

#hash_tree(path_or_dir = '', options = {}) ⇒ Hash<Object>

获取Hash格式的目录树形结构, 可用于直接to_json

:resource => {name: '', mtime: ''...,
:children => [
  => resource, :children => [...],
  => resource, :children => [...],
  ...
]

}

Examples:

puts bucket.hash_tree.to_json

Options Hash (options):

  • :depth (Integer)

    子目录深度,默认为5

Raises:


648
649
650
651
# File 'lib/cos/bucket.rb', line 648

def hash_tree(path_or_dir = '', options = {})
  dir = get_dir(path_or_dir)
  Tree.new(options.merge({path: dir})).to_hash
end

#list(path = '', options = {}) ⇒ Enumerator<Object> Also known as: ls

列出目录

Examples:

all = bucket.list
all.each do |o|
  if o.is_a?(COS::COSFile)
    puts "File: #{o.name} #{o.format_size}"
  else
    puts "Dir: #{o.name} #{o.created_at}"
  end
end

Options Hash (options):

  • :prefix (String)

    搜索前缀 如果填写prefix, 则列出含此前缀的所有文件及目录

  • :num (Integer)

    每页拉取的数量, 默认20条

  • :pattern (Symbol)

    获取方式 :dir_only 只获取目录, :file_only 只获取文件, 默认为 :both 全部获取

  • :order (Symbol)

    排序方式 :asc 正序, :desc 倒序 默认为 :asc

Raises:


168
169
170
# File 'lib/cos/bucket.rb', line 168

def list(path = '', options = {})
  Resource.new(self, path, options).to_enum
end

#list_count(path = '', options = {}) ⇒ Hash

获取list中的文件及目录个数

Options Hash (options):

  • :prefix (String)

    搜索前缀 如果填写prefix, 则计算含此前缀的所有文件及目录个数

Raises:


98
99
100
101
102
103
104
# File 'lib/cos/bucket.rb', line 98

def list_count(path = '', options = {})
  options = {}
  result  = client.api.list(path, options.merge({num: 1, bucket: bucket_name}))
  total   = result[:filecount] + result[:dircount]

  {total: total, files: result[:filecount], dirs: result[:dircount]}
end

#stat(path = '') ⇒ COSFile|COSDir

Note:

如查询根目录('/', '')可以获取到bucket信息, 返回COSDir

获取文件或目录信息

Examples:

puts bucket.stat('path1/file').name

Raises:


317
318
319
320
321
322
323
324
325
326
327
328
329
330
# File 'lib/cos/bucket.rb', line 317

def stat(path = '')
  data = client.api.stat(path, bucket: bucket_name)

  # 查询'/'获取的是bucket信息, 无name参数, 需要补全
  data[:name] = '' if data[:name].nil?

  if data[:filesize].nil?
    # 目录
    COSDir.new(data.merge({bucket: self, path: path}))
  else
    # 文件
    COSFile.new(data.merge({bucket: self, path: path}))
  end
end

#tree(path_or_dir = '', options = {}) ⇒ Hash

获取目录树形结构

:resource => resource,
:children => [
  {:resource => resource, :children => [...],
  => resource, :children => [...],
  ...
]

}

Examples:

tree = bucket.tree
puts tree[:resource].name
tree[:children].each do |r|
  puts r[:resource].name
end

Options Hash (options):

  • :depth (Integer)

    子目录深度,默认为5

Raises:


621
622
623
624
# File 'lib/cos/bucket.rb', line 621

def tree(path_or_dir = '', options = {})
  dir = get_dir(path_or_dir)
  Tree.new(options.merge({path: dir})).to_object
end

#update(path, biz_attr) ⇒ Object

更新文件及目录业务属性

Examples:

bucket.update('path1/file', 'i am the attr')

Raises:


341
342
343
# File 'lib/cos/bucket.rb', line 341

def update(path, biz_attr)
  client.api.update(path, biz_attr, bucket: bucket_name)
end

#upload(path_or_dir, file_name, file_src, options = {}) {|Float| ... } ⇒ COS::COSFile

上传文件, 大文件自动断点续传, 多线程上传

Examples:

file = bucket.upload('path', 'file1', '~/test/file1') do |p|
  puts "上传进度: #{(p*100).round(2)}%")
end
puts file.url

Options Hash (options):

  • :auto_create_folder (Boolean)

    自动创建远端目录

  • :min_slice_size (Integer)

    完整上传最小文件大小, 超过此大小将会使用分片多线程断点续传

  • :upload_retry (Integer)

    上传重试次数, 默认10

  • :biz_attr (String)

    目录属性, 业务端维护

  • :disable_cpt (Boolean)

    是否禁用checkpoint功能,如 果设置为true,则在上传的过程中不会写checkpoint文件,这意味着 上传失败后不能断点续传,而只能重新上传整个文件。如果这个值为 true,则:cpt_file会被忽略。

  • :threads (Integer)

    多线程上传线程数, 默认为10

  • :slice_size (Integer)

    设置分片上传时每个分片的大小 默认为3 MB, 目前服务端最大限制也为3MB。

  • :cpt_file (String)

    断点续传的checkpoint文件,如果 指定的cpt文件不存在,则会在file所在目录创建一个默认的cpt文件, 命名方式为:file.cpt,其中file是用户要上传的文件名。在上传的过 程中会不断更新此文件,成功完成上传后会删除此文件;如果指定的 cpt文件已存在,则从cpt文件中记录的点继续上传。

Yields:

  • (Float)

    上传进度百分比回调, 进度值是一个0-1之间的小数

Raises:


210
211
212
213
214
215
216
217
218
219
220
221
222
223
224
225
226
227
228
229
230
231
232
# File 'lib/cos/bucket.rb', line 210

def upload(path_or_dir, file_name, file_src, options = {}, &block)
  dir = get_dir(path_or_dir, options[:auto_create_folder])

  min_size    = options[:min_slice_size] || MIN_UPLOAD_SLICE_SIZE
  retry_times = options[:upload_retry] || DEFAULT_UPLOAD_RETRY

  options.merge!({bucket: bucket_name})
  file_src  = File.expand_path(file_src)
  file_size = File.size(file_src)

  retry_loop(retry_times) do
    if file_size > min_size
      # 分块上传
      client.api.upload_slice(dir.path, file_name, file_src, options, &block)
    else
      # 完整上传
      client.api.upload(dir.path, file_name, file_src, options)
    end
  end

  # 获取上传完成文件的状态, 只会返回<COSFile>
  stat(Util.get_list_path(dir.path, file_name, true))
end

#upload_all(path_or_dir, file_src_path, options = {}) {|Float| ... } ⇒ Array<COS::COSFile>

Note:

已存在的文件不会再次上传, 本地目录中的隐藏文件(已“.”开头的)不会上传 “.cpt”文件不会上传, 不会上传子目录

批量上传目录下的全部文件(不包含子目录)

Examples:

files = bucket.upload_all('path', '~/test') do |p|
  puts "上传进度: #{(p*100).round(2)}%")
end
files.each do |file|
  puts file.url
end

Options Hash (options):

  • :auto_create_folder (Boolean)

    自动创建远端目录

  • :min_slice_size (Integer)

    完整上传最小文件大小, 超过此大小将会使用分片多线程断点续传

  • :upload_retry (Integer)

    上传重试次数, 默认10

  • :biz_attr (String)

    目录属性, 业务端维护

  • :disable_cpt (Boolean)

    是否禁用checkpoint功能,如 果设置为true,则在上传的过程中不会写checkpoint文件,这意味着 上传失败后不能断点续传,而只能重新上传整个文件。如果这个值为 true,则:cpt_file会被忽略。

  • :threads (Integer)

    多线程上传线程数, 默认为10

  • :slice_size (Integer)

    设置分片上传时每个分片的大小 默认为3 MB, 目前服务端最大限制也为3MB。

  • :cpt_file (String)

    断点续传的checkpoint文件,如果 指定的cpt文件不存在,则会在file所在目录创建一个默认的cpt文件, 命名方式为:file.cpt,其中file是用户要上传的文件名。在上传的过 程中会不断更新此文件,成功完成上传后会删除此文件;如果指定的 cpt文件已存在,则从cpt文件中记录的点继续上传。

Yields:

  • (Float)

    上传进度百分比回调, 进度值是一个0-1之间的小数

Raises:


274
275
276
277
278
279
280
281
282
283
284
285
286
287
288
289
290
291
292
293
294
295
296
297
298
299
300
301
302
303
# File 'lib/cos/bucket.rb', line 274

def upload_all(path_or_dir, file_src_path, options = {}, &block)
  local_path = Util.get_local_path(file_src_path, false)
  uploaded   = []

  Dir.foreach(local_path) do |file|

    if !file.start_with?('.') and !file.end_with?('.cpt') and !File.directory?(file)
      logger.info("Begin to upload file >> #{file}")

      begin
        # 逐个上传
        uploaded << upload(path_or_dir, file, "#{local_path}/#{file}", options, &block)
      rescue => error
        # 跳过错误
        if options[:skip_error]
          logger.info("#{file} error skipped")
          next
        else
          # 终止上传抛出异常
          raise error
        end
      end

      logger.info("#{file} upload finished")
    end

  end

  uploaded
end

#url(path_or_file, options = {}) ⇒ String

Note:

私有读取的bucket会自动生成带签名的URL

获取文件可访问的URL

Examples:

puts bucket.url('path1/file1', https: true, cname: 'static.domain.com')

Options Hash (options):

  • :cname (String)

    在cos控制台设置的cname域名

  • :https (Boolean)

    是否生成https的URL

  • :expire_seconds (Integer)

    签名有效时间(秒,私有读取bucket时需要)

Raises:


441
442
443
444
445
446
447
448
449
450
451
452
453
454
455
456
457
458
459
460
461
462
463
464
465
466
467
468
# File 'lib/cos/bucket.rb', line 441

def url(path_or_file, options = {})

  file = get_file(path_or_file)

  url = file.access_url

  # 使用cname
  if options[:cname]
    host = URI.parse(url).host.downcase
    url.gsub!(host, options[:cname])
  end

  # 使用https
  if options[:https]
    url.gsub!('http://', 'https://')
  end

  if authority == 'eWRPrivate'
    # 私有读取的bucket自动生成带签名的URL
    sign = client.signature.multiple(
        bucket_name,
        options[:expire_seconds] || client.config.multiple_sign_expire)

    "#{url}?sign=#{sign}"
  else
    url
  end
end