Module: Laser::LexicalAnalysis

Defined in:
lib/laser/analysis/lexical_analysis.rb

Overview

This is a set of methods that get provided to Warnings so they can perform lexical analysis of their bodies. This module handles tokenizing only - not parse-trees.

Defined Under Namespace

Classes: Token

Instance Method Summary (collapse)

Instance Method Details

- (Array) find_keyword(*args)

Finds the first instance of a set of keywords in the body. If no text is given to scan, then the full content is scanned.

Parameters:

  • body (String) —

    (self.body) The first parameter is optional: the text to search. This defaults to the full text.

  • keyword (Symbol) —

    The rest of the arguments are keywords to search for. Any number of keywords may be specified.

Returns:

  • (Array) —

    the token in the form returned by Ripper. See #lex.



95
96
97
98
99
100
101
102
103
104
 
# File 'lib/laser/analysis/lexical_analysis.rb', line 95

def find_keyword(*args)
  body, list = _extract_token_search_args(args)
  list.map! {|x| x.to_s}
  lexed = lex(body)
  lexed.find.with_index do |tok, idx|
    is_keyword = tok.type == :on_kw && list.include?(tok.body)
    is_not_symbol = idx == 0 || lexed[idx-1].type != :on_symbeg
    is_keyword && is_not_symbol
  end
end

- (Array) find_token(*args)

Finds the first instance of a set of tokens in the body. If no text is given to scan, then the full content is scanned.

Parameters:

  • body (String) —

    (self.body) The first parameter is optional: the text to search. This defaults to the full text.

  • token (Symbol) —

    The rest of the arguments are tokens to search for. Any number of tokens may be specified.

Returns:

  • (Array) —

    the token in the form returned by Ripper. See #lex.



114
115
116
117
118
119
120
121
122
123
 
# File 'lib/laser/analysis/lexical_analysis.rb', line 114

def find_token(*args)
  body, list = _extract_token_search_args(args)
  # grr match comment with encoding in it
  lexed = lex(body)
  lexed.find.with_index do |tok, idx|
    is_token = list.include?(tok.type)
    is_not_symbol = idx == 0 || lexed[idx-1].type != :on_symbeg
    is_token && is_not_symbol
  end
end

- (Array<Array<Integer, Integer>, Symbol, String>) lex(body = self.body, token_class = Token)

Lexes the given text.

Parameters:

  • body (String) (defaults to: self.body) —

    (self.body) The text to lex

Returns:

  • (Array<Array<Integer, Integer>, Symbol, String>) —

    A set of tokens in Ripper's result format. Each token is an array of the form: [[1, token_position], token_type, token_text]. I'm not exactly clear on why the 1 is always there. At any rate - the result is an array of those tokens.



31
32
33
34
 
# File 'lib/laser/analysis/lexical_analysis.rb', line 31

def lex(body = self.body, token_class = Token)
  return [] if body =~ /^#.*encoding.*/
  Ripper.lex(body).map {|token| token_class.new(token) }
end

- (Array<String, String>) split_on_keyword(*args)

Splits the body into two halfs based on the first appearance of a keyword.

Examples:

split_on_keyword('x = 5 unless y == 2', :unless)
# => ['x = 5 ', 'unless y == 2']

Parameters:

  • body (String) —

    (self.body) The first parameter is optional: the text to search. This defaults to the full text.

  • token (Symbol) —

    The rest of the arguments are keywords to search for. Any number of keywords may be specified.

Returns:



135
136
137
138
139
 
# File 'lib/laser/analysis/lexical_analysis.rb', line 135

def split_on_keyword(*args)
  body, keywords = _extract_token_search_args(args)
  token = find_keyword(body, *keywords)
  return _split_body_with_raw_token(body, token)
end

- (Array<String, String>) split_on_token(*args)

Splits the body into two halfs based on the first appearance of a token.

Examples:

split_on_token('x = 5 unless y == 2', :on_kw)
# => ['x = 5 ', 'unless y == 2']

Parameters:

  • body (String) —

    (self.body) The first parameter is optional: the text to search. This defaults to the full text.

  • token (Symbol) —

    The rest of the arguments are tokens to search for. Any number of tokens may be specified.

Returns:



151
152
153
154
155
 
# File 'lib/laser/analysis/lexical_analysis.rb', line 151

def split_on_token(*args)
  body, tokens = _extract_token_search_args(args)
  token = find_token(body, *tokens)
  return _split_body_with_raw_token(body, token)
end

- (Object) text_between_token_positions(text, left, right, inclusive = :none)

Returns the text between two token positions. The token positions are in [line, column] format. The body, left, and right tokens must be provided, and optionally, you can override the inclusiveness of the text-between operation. It defaults to :none, for including neither the left nor right tokens in the result. You can pass :none, :left, :right, or :both.

Parameters:

  • body (String) —

    (self.body) The first parameter is optional: the text to search. This defaults to the full text.

  • left (Token) —

    the left token to get the text between

  • right (Token) —

    the right token to get the text between

  • inclusive (Symbol) (defaults to: :none) —

    should the :left, :right, :both, or :none tokens be included in the resulting text?

Returns:

  • the text between the two tokens within the text. This is necessary because the lexer provides [line, column] coordinates which is quite unfortunate.



51
52
53
54
55
56
57
58
59
60
61
62
63
 
# File 'lib/laser/analysis/lexical_analysis.rb', line 51

def text_between_token_positions(text, left, right, inclusive = :none)
  result = ""
  lines = text.lines.to_a
  left.line.upto(right.line) do |cur_line|
    line = lines[cur_line - 1]
    result << left.body if cur_line == left.line && (inclusive == :both || inclusive == :left)
    left_bound = cur_line == left.line ? left.col + left.body.size : 0
    right_bound = cur_line == right.line ? right.col - 1 : -1
    result << line[left_bound..right_bound]
    result << right.body if cur_line == right.line && (inclusive == :both || inclusive == :right)
  end
  result
end