The Functionalizer: Lossless Functional Decomposition for Subword Tokenization
The paper presents the Functionalizer, a pre-tokenization framework that decomposes orthographic variations (capitalization, diacritics, etc.) into reversible opcode/operand pairs encoded…