org.modeshape.common.text

## Interface TokenStream.Tokenizer

- All Known Implementing Classes:
    BasicSqlQueryParser.SqlTokenizer, CndTokenizer, DdlTokenStream.DdlTokenizer, FullTextSearchParser.TermTokenizer, TokenStream.BasicTokenizer, XPathParser.XPathTokenizer
  Enclosing class:
    TokenStream

      public static interface TokenStream.Tokenizer

  Interface for a Tokenizer component responsible for processing the characters in a `TokenStream.CharacterStream` and constructing the appropriate `TokenStream.Token` objects.

- - ### Method Summary

    Methods
    Modifier and Type | Method and Description
    --- | ---
    `void` | `tokenize(TokenStream.CharacterStream input, TokenStream.Tokens tokens)` Process the supplied characters and construct the appropriate `TokenStream.Token` objects.

- - ### Method Detail

    - #### tokenize

          void tokenize(TokenStream.CharacterStream input,
                      TokenStream.Tokens tokens)
                        throws ParsingException

      Process the supplied characters and construct the appropriate `TokenStream.Token` objects.
      Parameters:
        `input` - the character input stream; never null
        `tokens` - the factory for `TokenStream.Token` objects, which records the order in which the tokens are created
      Throws:
        `ParsingException` - if there is an error while processing the character stream (e.g., a quote is not closed, etc.)

Copyright © 2008-2014 JBoss, a division of Red Hat. All Rights Reserved.