Module: Laser::LexicalAnalysis

Defined in:
lib/laser/analysis/lexical_analysis.rb

Overview

This is a set of methods that get provided to Warnings so they can perform lexical analysis of their bodies. This module handles tokenizing only - not parse-trees.

Defined Under Namespace

Classes: Token

Instance Method Summary (collapse)

Instance Method Details

- (Array) find_keyword(*args)

Finds the first instance of a set of keywords in the body. If no text is given to scan, then the full content is scanned.



95
96
97
98
99
100
101
102
103
104
# File 'lib/laser/analysis/lexical_analysis.rb', line 95

def find_keyword(*args)
  body, list = _extract_token_search_args(args)
  list.map! {|x| x.to_s}
  lexed = lex(body)
  lexed.find.with_index do |tok, idx|
    is_keyword = tok.type == :on_kw && list.include?(tok.body)
    is_not_symbol = idx == 0 || lexed[idx-1].type != :on_symbeg
    is_keyword && is_not_symbol
  end
end

- (Array) find_token(*args)

Finds the first instance of a set of tokens in the body. If no text is given to scan, then the full content is scanned.



114
115
116
117
118
119
120
121
122
123
# File 'lib/laser/analysis/lexical_analysis.rb', line 114

def find_token(*args)
  body, list = _extract_token_search_args(args)
  # grr match comment with encoding in it
  lexed = lex(body)
  lexed.find.with_index do |tok, idx|
    is_token = list.include?(tok.type)
    is_not_symbol = idx == 0 || lexed[idx-1].type != :on_symbeg
    is_token && is_not_symbol
  end
end

- (Array<Array<Integer, Integer>, Symbol, String>) lex(body = self.body, token_class = Token)

Lexes the given text.



31
32
33
34
# File 'lib/laser/analysis/lexical_analysis.rb', line 31

def lex(body = self.body, token_class = Token)
  return [] if body =~ /^#.*encoding.*/
  Ripper.lex(body).map {|token| token_class.new(token) }
end

- (Array<String, String>) split_on_keyword(*args)

Splits the body into two halfs based on the first appearance of a keyword.

Examples:

split_on_keyword('x = 5 unless y == 2', :unless)
# => ['x = 5 ', 'unless y == 2']


135
136
137
138
139
# File 'lib/laser/analysis/lexical_analysis.rb', line 135

def split_on_keyword(*args)
  body, keywords = _extract_token_search_args(args)
  token = find_keyword(body, *keywords)
  return _split_body_with_raw_token(body, token)
end

- (Array<String, String>) split_on_token(*args)

Splits the body into two halfs based on the first appearance of a token.

Examples:

split_on_token('x = 5 unless y == 2', :on_kw)
# => ['x = 5 ', 'unless y == 2']


151
152
153
154
155
# File 'lib/laser/analysis/lexical_analysis.rb', line 151

def split_on_token(*args)
  body, tokens = _extract_token_search_args(args)
  token = find_token(body, *tokens)
  return _split_body_with_raw_token(body, token)
end

- (Object) text_between_token_positions(text, left, right, inclusive = :none)

Returns the text between two token positions. The token positions are in [line, column] format. The body, left, and right tokens must be provided, and optionally, you can override the inclusiveness of the text-between operation. It defaults to :none, for including neither the left nor right tokens in the result. You can pass :none, :left, :right, or :both.



51
52
53
54
55
56
57
58
59
60
61
62
63
# File 'lib/laser/analysis/lexical_analysis.rb', line 51

def text_between_token_positions(text, left, right, inclusive = :none)
  result = ""
  lines = text.lines.to_a
  left.line.upto(right.line) do |cur_line|
    line = lines[cur_line - 1]
    result << left.body if cur_line == left.line && (inclusive == :both || inclusive == :left)
    left_bound = cur_line == left.line ? left.col + left.body.size : 0
    right_bound = cur_line == right.line ? right.col - 1 : -1
    result << line[left_bound..right_bound]
    result << right.body if cur_line == right.line && (inclusive == :both || inclusive == :right)
  end
  result
end