Skip to content

Commit d92e39a

Browse files
committed
Add Cow-based internal tokenizer functions for zero-copy preparation
Add internal _borrowed() functions that return Cow<\'a, str> to prepare for zero-copy tokenization. When the source string needs no transformation (no escaping), return Cow::Borrowed. When transformation is required, return Cow::Owned. The Token enum still uses String, so borrowed values are converted via to_owned() for now. This maintains API compatibility while preparing the codebase for a future refactor where Token can hold borrowed strings. Optimized: comments, quoted strings, dollar-quoted strings, quoted identifiers.
1 parent 82c6657 commit d92e39a

File tree

1 file changed

+315
-95
lines changed

1 file changed

+315
-95
lines changed

0 commit comments

Comments
 (0)