Longest-match Tokenizer

Implement tokenize(text, vocab) for a simplified tokenizer. At each character position, choose the longest token from vocab that matches the remaining text.

Matching is greedy, not globally optimal. For vocabulary {"ab", "abc", "cd"}, the input "abcd" chooses "abc" and then fails at position 3; it must not backtrack to the valid segmentation ["ab", "cd"].

Requirements

• Return tokens in order.
• Ignore ASCII whitespace between tokens (space, tab, newline, carriage return, form feed, vertical tab). Python str.isspace() is also acceptable if it skips additional Unicode whitespace.
• Prefer longest match when multiple tokens match.
• Raise ValueError with the failing position when no token matches.
• Raise ValueError if vocab contains an empty string.
• Avoid repeated full-string slicing in the hot loop.

Example

At position 0, both the and there match. Choosing there leaves fore as the next longest match.

Constraints

• Treat empty vocab entries as invalid input (raise), not as a match.
• Input text is small enough for a trie or length-sorted scan.
• Use standard-library Python only.