Introduction
Sometimes you need to use strtok in MFC projects, but it has
two big limitations:
- Can't use it to tokenize more than one string at the time.
- It's not easy to use safely with
CStrings
To overcome these limitations I wrote CTokenizer.
Usage
It's quite easy to use as the following code shows:
CTokenizer tok(_T("A-B+C*D-E"), _T("-+"));
CString cs;
while(tok.Next(cs))
TRACE2("Token: '%s', Tail: '%s'\n", (LPCTSTR)cs, (LPCTSTR)tok.Tail());
The preceding code produces the following output:
Token: 'A', Tail: 'B+C*D-E'
Token: 'B', Tail: 'C*D-E'
Token: 'C*D', Tail: 'E'
Token: 'E', Tail: ''
As you can see, it's no rocket science! It's a very simple and handy class
To summarize:
- Create an object of type
CTokenizer, passing as arguments the
string to tokenize and the delimiters
- Call
Next() until it returns false
- If you need to change the delimiters, call
SetDelimiters()
- If you need the un-tokenized part of the string, call
Tail()
Updates
- 10/22/2001 Using
std::bitset instead of std::vector.
Reduces memory requirements and increases performance.
- 10/22/2001 Fixed bug related to ASCII characters above 127. Reported by John Simpsons
