Using the StreamTokenizer class - A Java Code Example

This java code example shows how you can use the StreamTokenizer class to count words and numbers in a file.
In the example we open a stream to a file with help of the FileReader class.
We construct the StreamTokenizer object with this stream as argument and start looping through it.
The 'nextToken' method returns an int which indicates of what type the token is and we can match the int value to one of the static variables of the StreamTokenizer class to determine of what type the current token is.
The iterations continues until the end of file token is reached. For each token we check the tokenizers variable 'ttype', which holds the current token type, against the static variables to see if it is a number or a word and increment count variables accordingly.
So if the file content would look like this (without the quotes):

"5 ants is more than 4 elephants"

The output would be:

"Number of words in file: 5"
"Number of numbers in file: 2"

import java.io.FileNotFoundException;
import java.io.FileReader;
import java.io.IOException;
import java.io.StreamTokenizer;

/**
*
* @author javadb.com
*/
public class Main {

    /**
     * Example method for using the StreamTokenizer class
     */
    public void countWordsAndNumbers(String filename) {

        StreamTokenizer sTokenizer = null;
        int wordCount = 0, numberCount = 0;

        try {

            sTokenizer = new StreamTokenizer(new FileReader(filename));

            while (sTokenizer.nextToken() != StreamTokenizer.TT_EOF) {

                if (sTokenizer.ttype == StreamTokenizer.TT_WORD)
                    wordCount++;
                else if (sTokenizer.ttype == StreamTokenizer.TT_NUMBER)
                    numberCount++;
            }

            System.out.println("Number of words in file: " + wordCount);
            System.out.println("Number of numbers in file: " + numberCount);

        } catch (FileNotFoundException ex) {
            ex.printStackTrace();
        } catch (IOException ex) {
            ex.printStackTrace();
        }
    }

    /**
     * @param args the command line arguments
     */
    public static void main(String[] args) {
        new Main().countWordsAndNumbers("myFile.txt");
    }
}

Search for code examples on this site