The StringTokenizer class

Summary:

Basics
Examples

Basics

Purpose:

The StringTokenizer class is designed to parse a string to extract tokens according to delimiters.

Syntax:

base.StringTokenizer

Methods:

Class Methods
Name	Description
`create( src` `STRING` `, delim` `STRING` `)` `RETURNING` `base.StringTokenizer`	Returns a StringTokenizer object prepared to parse the src source string according to the delim delimiters. The delim parameter is a string that can hold one or more delimiters.
`createExt( src` `STRING` `, delim` `STRING` `, esc` `STRING, nullsINTEGER)` `RETURNING` `base.StringTokenizer`	Same as create(), except for additional options. The esc parameter defines an escape character for the delimiter. The nulls parameter indicates if empty tokens are taken into account.
Object Methods
Name	Description
`countTokens()` `RETURNING INTEGER`	Returns the number of tokens left to be returned.
`hasMoreTokens()` `RETURNING INTEGER`	Returns TRUE if there are more tokens to return.
`nextToken()` `RETURNING STRING`	Parses the string and returns the next token.

Usage:

The StringTokenizer built-in class is provided to split a source string into tokens, according to delimiters. The following code creates a tokenizer that will generate 3 tokens with the values "aaa", "bbb", "ccc":

01 DEFINE tok base.StringTokenizer
02 LET tok = base.StringTokenizer.create("aaa|bbb|ccc","|")

The StringTokenizer can take a unique or multiple delimiters into account. A delimiter is always one character long. In the following example, 3 delimiters are used, and 4 tokens are extracted:

01 DEFINE tok base.StringTokenizer
02 LET tok = base.StringTokenizer.create("aaa|bbb;ccc+ddd","|+;")

If you create a StringTokenizer with the create(src,delim) method, the empty tokens are not taken into account, and not escape character is defined for the delimiters:

No escape character can be used.
The nextToken() method will never return NULL strings.
In the source string, leading and trailing delimiters or the amount of delimiters between two tokens do not affect the number of tokens.

If you create a StringTokenizer with the createExt(src,delim,esc,nulls) method, you can configure the tokenizer:

When passing a character to the esc parameter, the delimiters can be escaped in the source string.

When passing TRUE to the nulls parameter, the empty tokens are taken into account:

The nextToken() method might return NULL strings.
In the source string, leading and trailing delimiters or the amount of delimiters between two tokens affects the number of tokens.

Note that when you want to specify a backslash as a delimiter, you must use double backslashes in both the source string and as the delimiter, as shown in Example 3 below.

Examples

Example 1: Split a UNIX directory path

01 MAIN
02   DEFINE tok base.StringTokenizer
03   LET tok = base.StringTokenizer.create("/home/tomy","/")
04   WHILE tok.hasMoreTokens()
05     DISPLAY tok.nextToken()
06   END WHILE
07 END MAIN

Example 2: Taking escaped delimiters and NULL tokens into account

01 MAIN
02   DEFINE tok base.StringTokenizer
03   LET tok = base.StringTokenizer.createExt("||\\|aaa||bbc|","|","\\",TRUE)
04   WHILE tok.hasMoreTokens()
05     DISPLAY tok.nextToken()
06   END WHILE
07 END MAIN

Example 3: Specifying a backslash as the delimiter

01 MAIN
02   DEFINE tok base.StringTokenizer
03   LET tok = base.StringTokenizer.create("C:\\My Documents\\My Pictures","\\")
04   WHILE tok.hasMoreTokens()
05     DISPLAY tok.nextToken()
06   END WHILE
07 END MAIN