# Poco

# class StreamTokenizer

Library: Foundation
Package: Streams
Header: Poco/StreamTokenizer.h

## Description

A stream tokenizer splits an input stream into a sequence of tokens of different kinds. Various token kinds can be registered with the tokenizer.

## Member Summary

Member Functions: addToken, attachToStream, next

## Constructors

### StreamTokenizer

StreamTokenizer();

Creates a StreamTokenizer with no attached stream.

### StreamTokenizer

StreamTokenizer(
std::istream & istr
);

Creates a StreamTokenizer with no attached stream.

## Destructor

### ~StreamTokenizer

virtual ~StreamTokenizer();

Destroys the StreamTokenizer and deletes all registered tokens.

## Member Functions

### addToken

void addToken(
Token * pToken
);

Adds a token class to the tokenizer. The tokenizer takes ownership of the token and deletes it when no longer needed. Comment and whitespace tokens will be marked as ignorable, which means that next() will not return them.

### addToken

void addToken(
Token * pToken,
bool ignore
);

Adds a token class to the tokenizer. The tokenizer takes ownership of the token and deletes it when no longer needed. If ignore is true, the token will be marked as ignorable, which means that next() will not return it.

### attachToStream

void attachToStream(
std::istream & istr
);

Attaches the tokenizer to an input stream.

### next

const Token * next();

Extracts the next token from the input stream. Returns a pointer to an EOFToken if there are no more characters to read. Returns a pointer to an InvalidToken if an invalid character is encountered. If a token is marked as ignorable, it will not be returned, and the next token will be examined. Never returns a nullptr pointer. You must not delete the token returned by next().