Overview | Package | Class | Use | Tree | Deprecated | Index | Help
PREV CLASS NEXT CLASS | FRAMES NO FRAMES
SUMMARY: NESTED | FIELD | CONSTR | METHOD | DETAIL: FIELD | CONSTR | METHOD

## org.modeshape.common.text
Interface TokenStream.Tokenizer

All Known Implementing Classes:
  CndTokenizer, DdlTokenStream.DdlTokenizer, FullTextSearchParser.TermTokenizer, SqlQueryParser.SqlTokenizer, TokenStream.BasicTokenizer

Enclosing class:
  TokenStream

    public static interface TokenStream.Tokenizer

Interface for a Tokenizer component responsible for processing the characters in a `TokenStream.CharacterStream` and constructing the appropriate `TokenStream.Token` objects.

Method Summary
`void` | `tokenize(TokenStream.CharacterStream input, TokenStream.Tokens tokens)` Process the supplied characters and construct the appropriate `TokenStream.Token` objects.

Method Detail

### tokenize

    void tokenize(TokenStream.CharacterStream input,
                  TokenStream.Tokens tokens)
                  throws ParsingException

  Process the supplied characters and construct the appropriate `TokenStream.Token` objects.

  Parameters:
    `input` - the character input stream; never null
    `tokens` - the factory for `TokenStream.Token` objects, which records the order in which the tokens are created
  Throws:
    `ParsingException` - if there is an error while processing the character stream (e.g., a quote is not closed, etc.)

Overview | Package | Class | Use | Tree | Deprecated | Index | Help
PREV CLASS NEXT CLASS | FRAMES NO FRAMES
SUMMARY: NESTED | FIELD | CONSTR | METHOD | DETAIL: FIELD | CONSTR | METHOD

Copyright © 2008-2012 JBoss, a division of Red Hat. All Rights Reserved.