Class: MiGA::Dataset

Inherits:
MiGA
  • Object
show all
Includes:
DatasetResult
Defined in:
lib/miga/dataset.rb

Overview

Dataset representation in MiGA.

Constant Summary collapse

@@RESULT_DIRS =
{
  # Preprocessing
  raw_reads: "01.raw_reads", trimmed_reads: "02.trimmed_reads",
  read_quality: "03.read_quality", trimmed_fasta: "04.trimmed_fasta",
  assembly: "05.assembly", cds: "06.cds",
  # Annotation
  essential_genes: "07.annotation/01.function/01.essential",
  ssu: "07.annotation/01.function/02.ssu",
  mytaxa: "07.annotation/02.taxonomy/01.mytaxa",
  mytaxa_scan: "07.annotation/03.qa/02.mytaxa_scan",
  # Mapping
  mapping_on_contigs: "08.mapping/01.read-ctg",
  mapping_on_genes: "08.mapping/02.read-gene",
  # Distances (for single-species datasets)
  distances: "09.distances"
}
@@KNOWN_TYPES =
{
  genome: {description: "The genome from an isolate.", multi: false},
  metagenome: {description: "A metagenome (excluding viromes).",
    multi: true},
  virome: {description: "A viral metagenome.", multi: true},
  scgenome: {description: "A genome from a single cell.", multi: false},
  popgenome: {description: "The genome of a population (including " +
    "microdiversity).", :multi=>false}
}
@@PREPROCESSING_TASKS =
[:raw_reads, :trimmed_reads, :read_quality,
:trimmed_fasta, :assembly, :cds, :essential_genes, :ssu, :mytaxa,
:mytaxa_scan, :distances]
@@EXCLUDE_NOREF_TASKS =

Tasks to be excluded from query datasets.

[:mytaxa_scan]
@@ONLY_NONMULTI_TASKS =

Tasks to be executed only in datasets that are not multi-organism. These tasks are ignored for multi-organism datasets or for unknown types.

[:mytaxa_scan, :distances]
@@ONLY_MULTI_TASKS =

Tasks to be executed only in datasets that are multi-organism. These tasks are ignored for single-organism datasets or for unknwon types.

[:mytaxa]

Constants included from MiGA

CITATION, VERSION, VERSION_DATE, VERSION_NAME

Instance Attribute Summary collapse

Class Method Summary collapse

Instance Method Summary collapse

Methods inherited from MiGA

CITATION, DEBUG, DEBUG_OFF, DEBUG_ON, DEBUG_TRACE_OFF, DEBUG_TRACE_ON, FULL_VERSION, LONG_VERSION, VERSION, VERSION_DATE, initialized?, #result_files_exist?, root_path, tabulate

Constructor Details

#initialize(project, name, is_ref = true, metadata = {}) ⇒ Dataset

Create a MiGA::Dataset object in a project MiGA::Project with a uniquely identifying name. is_ref indicates if the dataset is to be treated as reference (true, default) or query (false). Pass any additional metadata as a Hash.


101
102
103
104
105
106
107
108
109
# File 'lib/miga/dataset.rb', line 101

def initialize(project, name, is_ref=true, ={})
  raise "Invalid name '#{name}', please use only alphanumerics and " +
    "underscores." unless name.miga_name?
  @project = project
  @name = name
  [:ref] = is_ref
  @metadata = MiGA::Metadata.new(project.path + "/metadata/" + name + ".json",
    )
end

Instance Attribute Details

#metadataObject (readonly)

MiGA::Metadata with information about the dataset.


94
95
96
# File 'lib/miga/dataset.rb', line 94

def 
  @metadata
end

#nameObject (readonly)

Datasets are uniquely identified by name in a project.


90
91
92
# File 'lib/miga/dataset.rb', line 90

def name
  @name
end

#projectObject (readonly)

MiGA::Project that contains the dataset.


86
87
88
# File 'lib/miga/dataset.rb', line 86

def project
  @project
end

Class Method Details

.exist?(project, name) ⇒ Boolean

Does the project already have a dataset with that name?

Returns:

  • (Boolean)

72
73
74
# File 'lib/miga/dataset.rb', line 72

def self.exist?(project, name)
  File.exist? project.path + "/metadata/" + name + ".json"
end

.INFO_FIELDSObject

Standard fields of metadata for datasets.


78
79
80
# File 'lib/miga/dataset.rb', line 78

def self.INFO_FIELDS
  %w(name created updated type ref user description comments)
end

.KNOWN_TYPESObject

Supported dataset types.


38
# File 'lib/miga/dataset.rb', line 38

def self.KNOWN_TYPES ; @@KNOWN_TYPES end

.PREPROCESSING_TASKSObject

Returns an Array of tasks to be executed before project-wide tasks.


51
# File 'lib/miga/dataset.rb', line 51

def self.PREPROCESSING_TASKS ; @@PREPROCESSING_TASKS ; end

.RESULT_DIRSObject

Directories containing the results from dataset-specific tasks.


18
# File 'lib/miga/dataset.rb', line 18

def self.RESULT_DIRS ; @@RESULT_DIRS end

Instance Method Details

#add_result(result_type, save = true) ⇒ Object

Look for the result with symbol key result_type and register it in the dataset. If save is false, it doesn't register the result, but it still returns a result if the expected files are complete. Returns MiGA::Result or nil.


178
179
180
181
182
183
184
185
186
187
# File 'lib/miga/dataset.rb', line 178

def add_result(result_type, save=true)
  return nil if @@RESULT_DIRS[result_type].nil?
  base = project.path + "/data/" + @@RESULT_DIRS[result_type] +
    "/" + name
  return MiGA::Result.load("#{base}.json") unless save
  return nil unless result_files_exist?(base, ".done")
  r = self.send("add_result_#{result_type}", base)
  r.save unless r.nil?
  r
end

#done_preprocessing?(save = false) ⇒ Boolean

Are all the dataset-specific tasks done? Passes save to #add_result.

Returns:

  • (Boolean)

222
223
224
# File 'lib/miga/dataset.rb', line 222

def done_preprocessing?(save=false)
  !first_preprocessing(save).nil? and next_preprocessing(save).nil?
end

#each_result(&blk) ⇒ Object

For each result executes the 2-ary blk block: key symbol and MiGA::Result.


167
168
169
170
171
# File 'lib/miga/dataset.rb', line 167

def each_result(&blk)
  @@RESULT_DIRS.keys.each do |k|
    blk.call(k, result(k)) unless result(k).nil?
  end
end

#first_preprocessing(save = false) ⇒ Object

Returns the key symbol of the first registered result (sorted by the execution order). This typically corresponds to the result used as the initial input. Passes save to #add_result.


193
194
195
# File 'lib/miga/dataset.rb', line 193

def first_preprocessing(save=false)
  @@PREPROCESSING_TASKS.find{ |t| not add_result(t, save).nil? }
end

#ignore_task?(task) ⇒ Boolean

Should I ignore task for this dataset?

Returns:

  • (Boolean)

214
215
216
217
218
# File 'lib/miga/dataset.rb', line 214

def ignore_task?(task)
  ( (@@EXCLUDE_NOREF_TASKS.include?(task) and not is_ref?) or
    (@@ONLY_MULTI_TASKS.include?(task) and not is_multi?) or
    (@@ONLY_NONMULTI_TASKS.include?(task) and not is_nonmulti?))
end

#infoObject

Get standard metadata values for the dataset as Array.


129
130
131
132
133
# File 'lib/miga/dataset.rb', line 129

def info
  MiGA::Dataset.INFO_FIELDS.map do |k|
    (k=="name") ? self.name : self.[k.to_sym]
  end
end

#is_multi?Boolean

Is this dataset known to be multi-organism?

Returns:

  • (Boolean)

141
142
143
144
# File 'lib/miga/dataset.rb', line 141

def is_multi?
  return false if self.[:type].nil?
  @@KNOWN_TYPES[self.[:type]][:multi]
end

#is_nonmulti?Boolean

Is this dataset known to be single-organism?

Returns:

  • (Boolean)

148
149
150
151
# File 'lib/miga/dataset.rb', line 148

def is_nonmulti?
  return false if self.[:type].nil?
  !@@KNOWN_TYPES[self.[:type]][:multi]
end

#is_ref?Boolean

Is this dataset a reference?

Returns:

  • (Boolean)

137
# File 'lib/miga/dataset.rb', line 137

def is_ref? ; !!self.[:ref] ; end

#next_preprocessing(save = false) ⇒ Object

Returns the key symbol of the next task that needs to be executed. Passes save to #add_result.


200
201
202
203
204
205
206
207
208
209
210
# File 'lib/miga/dataset.rb', line 200

def next_preprocessing(save=false)
  after_first = false
  first = first_preprocessing(save)
  return nil if first.nil?
  @@PREPROCESSING_TASKS.each do |t|
    next if ignore_task? t
    return t if after_first and add_result(t, save).nil?
    after_first = (after_first or (t==first))
  end
  nil
end

#profile_advance(save = false) ⇒ Object

Returns an array indicating the stage of each task (sorted by execution order). The values are integers:

  • 0 for an undefined result (a task before the initial input).

  • 1 for a registered result (a completed task).

  • 2 for a queued result (a task yet to be executed).

It passes save to #add_result


233
234
235
236
237
238
239
240
241
242
243
244
245
# File 'lib/miga/dataset.rb', line 233

def profile_advance(save=false)
  first_task = first_preprocessing(save)
  return Array.new(@@PREPROCESSING_TASKS.size, 0) if first_task.nil?
  adv = []
  state = 0
  next_task = next_preprocessing(save)
  @@PREPROCESSING_TASKS.each do |task|
    state = 1 if first_task==task
    state = 2 if !next_task.nil? and next_task==task
    adv << state
  end
  adv
end

#remove!Object

Delete the dataset with all it's contents (including results) and returns nil.


122
123
124
125
# File 'lib/miga/dataset.rb', line 122

def remove!
  self.results.each{ |r| r.remove! }
  self..remove!
end

#result(k) ⇒ Object

Get the result MiGA::Result in this dataset identified by the symbol k.


155
156
157
158
159
# File 'lib/miga/dataset.rb', line 155

def result(k)
  return nil if @@RESULT_DIRS[k.to_sym].nil?
  MiGA::Result.load(project.path + "/data/" + @@RESULT_DIRS[k.to_sym] +
    "/" + name + ".json")
end

#resultsObject

Get all the results (Array of MiGA::Result) in this dataset.


163
# File 'lib/miga/dataset.rb', line 163

def results ; @@RESULT_DIRS.keys.map{ |k| result k }.compact ; end

#saveObject

Save any changes you've made in the dataset.


113
114
115
116
117
# File 'lib/miga/dataset.rb', line 113

def save
  self.[:type] = :metagenome if ![:tax].nil? and
    ![:tax][:ns].nil? and [:tax][:ns]=="COMMUNITY"
  self..save
end