Module: RDoc::TokenStream

Defined in:
lib/rdoc/token_stream.rb

Overview

A TokenStream is a list of tokens, gathered during the parse of some entity (say a method). Entities populate these streams by being registered with the lexer. Any class can collect tokens by including TokenStream. From the outside, you use such an object by calling the start_collecting_tokens method, followed by calls to add_token and pop_token.

Instance Method Summary (collapse)

Instance Method Details

- (Object) add_tokens(*tokens) Also known as: add_token

Adds tokens to the collected tokens



13
14
15
 
# File 'lib/rdoc/token_stream.rb', line 13

def add_tokens(*tokens)
  tokens.flatten.each { |token| @token_stream << token }
end

- (Object) collect_tokens Also known as: start_collecting_tokens

Starts collecting tokens



22
23
24
 
# File 'lib/rdoc/token_stream.rb', line 22

def collect_tokens
  @token_stream = []
end

- (Object) pop_token

Remove the last token from the collected tokens



31
32
33
 
# File 'lib/rdoc/token_stream.rb', line 31

def pop_token
  @token_stream.pop
end

- (Object) token_stream

Current token stream



38
39
40
 
# File 'lib/rdoc/token_stream.rb', line 38

def token_stream
  @token_stream
end

- (Object) tokens_to_s

Returns a string representation of the token stream



45
46
47
 
# File 'lib/rdoc/token_stream.rb', line 45

def tokens_to_s
  token_stream.map { |token| token.text }.join ''
end