void nb_tokenize(char *text, unsigned char *tp);
nb_tokenize tokenizes the input text into the token stream tokens.