C++ Boost

Token Iterator

template <
	class TokenizerFunc = char_delimiters_separator<char>, 
	class Iterator = std::string::const_iterator,
	class Type = std::string
>
class token_iterator_generator 
template<class Type, class Iterator, class TokenizerFunc>
typename token_iterator_generator<TokenizerFunc,Iterator,Type>::type 
make_token_iterator(Iterator begin, Iterator end,const TokenizerFunc& fun)

The token iterator serves to provide an iterator view of the tokens in a parsed sequence.

Example

/// simple_example_5.cpp
#include<iostream>
#include<boost/token_iterator.hpp>
#include<string>

int main(){
   using namespace std;
   using namespace boost;
   string s = "12252001";
   int offsets[] = {2,2,4};
   offset_separator f(offsets, offsets+3);
   typedef token_iterator_generator<offset_separator>::type Iter;
   Iter beg = make_token_iterator<string>(s.begin(),s.end(),f);
   Iter end = make_token_iterator<string>(s.end(),s.end(),f); 
   // The above statement could also have been what is below
   // Iter end;
   for(;beg!=end;++beg){
     cout << *beg << "\n";
   }
}


 

Template Parameters

Parameter Description
TokenizerFunc The TokenizerFunction used to parse the sequence.
Iterator The type of the iterator the specifies the sequence.
Type The type of the token, typically string.

Model of

The category of Iterator, up to and including Forward Iterator. Anything higher will get scaled down to Forward Iterator.

Related Types

Type

Remarks

token_iterator_generator::type The type of the token iterator.

Creation

template<class Type, class Iterator, class TokenizerFunc>
typename token_iterator_generator<TokenizerFunc,Iterator,Type>::type 
make_token_iterator(Iterator begin, Iterator end,const TokenizerFunc& fun)

Parameter

Description

begin The beginning of the sequence to be parsed.
end Past the end of the sequence to be parsed.
fun A functor that is a model of TokenizerFunction

 


© Copyright John R. Bandela 2001. Permission to copy, use, modify, sell and distribute this document is granted provided this copyright notice appears in all copies. This document is provided "as is" without express or implied warranty, and with no claim as to its suitability for any purpose.