Class: ETL::Processor::SftpDownloaderProcessor

Inherits:
Processor show all
Defined in:
lib/etl/processor/sftp_downloader_processor.rb

Overview

Custom processor to download files via SFTP

Instance Attribute Summary (collapse)

Instance Method Summary (collapse)

Constructor Details

- (SftpDownloaderProcessor) initialize(control, configuration)

configuration options include:

  • host - hostname or IP address of FTP server (required)

  • port - port number for FTP server (default: 22)

  • remote_dir - remote path on FTP server (default: /)

  • files - list of files to download from FTP server (default: [])

  • username - username for FTP server authentication (default: anonymous)

  • password - password for FTP server authentication (default: nil)

  • local_dir - local output directory to save downloaded files (default: '')

As an example you might write something like the following in your control process file:

pre_process :sftp_downloader, {
  :host => 'sftp.sec.gov',
  :path => 'edgar/Feed/2007/QTR2',
  :files => ['20070402.nc.tar.gz', '20070403.nc.tar.gz', '20070404.nc.tar.gz', 
             '20070405.nc.tar.gz', '20070406.nc.tar.gz'],
  :local_dir => '/data/sec/2007/04',
}

The above example will anonymously download via SFTP the first week's worth of SEC filing feed data from the second quarter of 2007 and download the files to the local directory /data/sec/2007/04.



33
34
35
36
37
38
39
40
41
 
# File 'lib/etl/processor/sftp_downloader_processor.rb', line 33

def initialize(control, configuration)
  @host = configuration[:host]
  @port = configuration[:port] || 22
  @remote_dir = configuration[:remote_dir] || '/'
  @files = configuration[:files] || []
  @username = configuration[:username] || 'anonymous'
  @password = configuration[:password]
  @local_dir = configuration[:local_dir] || ''
end

Instance Attribute Details

- (Object) files (readonly)

Returns the value of attribute files



10
11
12
 
# File 'lib/etl/processor/sftp_downloader_processor.rb', line 10

def files
  @files
end

- (Object) host (readonly)

Returns the value of attribute host



7
8
9
 
# File 'lib/etl/processor/sftp_downloader_processor.rb', line 7

def host
  @host
end

- (Object) local_dir (readonly)

Returns the value of attribute local_dir



12
13
14
 
# File 'lib/etl/processor/sftp_downloader_processor.rb', line 12

def local_dir
  @local_dir
end

- (Object) port (readonly)

Returns the value of attribute port



8
9
10
 
# File 'lib/etl/processor/sftp_downloader_processor.rb', line 8

def port
  @port
end

- (Object) remote_dir (readonly)

Returns the value of attribute remote_dir



9
10
11
 
# File 'lib/etl/processor/sftp_downloader_processor.rb', line 9

def remote_dir
  @remote_dir
end

- (Object) username (readonly)

Returns the value of attribute username



11
12
13
 
# File 'lib/etl/processor/sftp_downloader_processor.rb', line 11

def username
  @username
end

Instance Method Details

- (Object) process 



43
44
45
46
47
48
49
 
# File 'lib/etl/processor/sftp_downloader_processor.rb', line 43

def process
  Net::SFTP.start(@host, @username, {:port => @port, :password => @password}) do |conn|
    @files.each do |f|
      conn.download!(remote_file(f), local_file(f))
    end
  end
end