The StringTokenizer class

Summary:

Syntax
Method s
Usage
Examples

Syntax

The StringTokenizer class is designed to parse a string to extract tokens according to delimiters.

Syntax:

base.StringTokenizer

Methods:

Class Methods
Name	Description
`create( src STRING, delim STRING ) RETURNING base.StringTokenizer`	Returns a StringTokenizer object prepared to parse the src source string according to the delim delimiters. The delim parameter is a string that can hold one or more delimiters.
`createExt( src STRING, delim STRING, esc STRING, nulls INTEGER ) RETURNING base.StringTokenizer`	Same as create(), except for additional options. The esc parameter defines an escape character for the delimiter. The nulls parameter indicates if empty tokens are taken into account.
Object Methods
Name	Description
`countTokens() RETURNING INTEGER`	Returns the number of tokens left to be returned.
`hasMoreTokens() RETURNING INTEGER`	Returns TRUE if there are more tokens to return.
`nextToken() RETURNING STRING`	Parses the string and returns the next token.

The StringTokenizer built-in class is provided to split a source string into tokens, according to delimiters. The following code uses the base.StringTokenizer.create() method to create a StringTokenizer that will generate 3 tokens with the values "aaa", "bbb", "ccc":

01 DEFINE tok base.StringTokenizer
02 LET tok = base.StringTokenizer.create("aaa|bbb|ccc","|")

The StringTokenizer can take a unique or multiple delimiters into account. A delimiter is always one character long. In the following example, 3 delimiters are used, and 4 tokens are extracted:

01 DEFINE tok base.StringTokenizer
02 LET tok = base.StringTokenizer.create("aaa|bbb;ccc+ddd","|+;")

If you create a StringTokenizer with the base.StringTokenizer.create(src,delim) method, the empty tokens are not taken into account, and no escape character is defined for the delimiters:

No escape character can be used.
The nextToken() method will never return NULL strings.
In the source string, leading and trailing delimiters or the amount of delimiters between two tokens do not affect the number of tokens.

If you create a StringTokenizer with the base.StringTokenizer.reateExt(src,delim,esc,nulls) method, you can configure the StringTokenizer:

When passing a character to the esc parameter, the delimiters can be escaped in the source string.

When passing TRUE to the nulls parameter, the empty tokens are taken into account:

The nextToken() method might return NULL strings.
In the source string, leading and trailing delimiters or the amount of delimiters between two tokens affects the number of tokens.

Note that when you want to specify a backslash as a delimiter, you must use double backslashes in both the source string and as the delimiter, as shown in Example 3 below.

The countTokens() method counts the number of tokens left to be returned.

The hasMoreTokens() method returns TRUE if there are more tokens to return.

The nextToken() method parses the string and returns the next token.

Examples

Example 1: Split a UNIX directory path

01 MAIN
02   DEFINE tok base.StringTokenizer
03   LET tok = base.StringTokenizer.create("/home/tomy","/")
04   WHILE tok.hasMoreTokens()
05     DISPLAY tok.nextToken()
06   END WHILE
07 END MAIN

Example 2: Taking escaped delimiters and NULL tokens into account

01 MAIN
02   DEFINE tok base.StringTokenizer
03   LET tok = base.StringTokenizer.createExt("||\\|aaa||bbc|","|","\\",TRUE)
04   WHILE tok.hasMoreTokens()
05     DISPLAY tok.nextToken()
06   END WHILE
07 END MAIN

Example 3: Specifying a backslash as the delimiter

01 MAIN
02   DEFINE tok base.StringTokenizer
03   LET tok = base.StringTokenizer.create("C:\\My Documents\\My Pictures","\\")
04   WHILE tok.hasMoreTokens()
05     DISPLAY tok.nextToken()
06   END WHILE
07 END MAIN