Longest-match Tokenizer

Implement tokenize(text, vocab) for a simplified tokenizer. At each character position, choose the longest token from vocab that matches the remaining text.

Matching is greedy, not globally optimal. For vocabulary {"ab", "abc", "cd"}, the input "abcd" chooses "abc" and then fails at position 3; it must not backtrack to the valid segmentation ["ab", "cd"].

Requirements

• Return tokens in order.
• Ignore ASCII whitespace between tokens (space, tab, newline, carriage return, form feed, vertical tab). Python str.isspace() is also acceptable if it skips additional Unicode whitespace.
• Prefer longest match when multiple tokens match.
• Raise ValueError with the failing position when no token matches.
• Positions are zero-based Unicode code-point offsets in the original text, including skipped whitespace. Matching is exact and case-sensitive; don't normalize Unicode or casing. NUL (\0) is an ordinary character and can appear in a token.
• Raise ValueError if vocab contains an empty string.
• Avoid repeated full-string slicing in the hot loop.
• Skip whitespace before beginning each token search. Whitespace inside a matching vocabulary token remains literal: tokenize("a b", {"a b", "a", "b"}) returns ["a b"].

Example

At position 0, both the and there match. Choosing there leaves fore as the next longest match.

vocab = {"the", "there", "for", "fore"}
assert tokenize("therefore", vocab) == ["there", "fore"]

Constraints

• Treat empty vocab entries as invalid input (raise), not as a match.
• Input text is small enough for a trie or length-sorted scan.
• Use standard-library Python only.