You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
Add Cow-based internal tokenizer functions for zero-copy preparation
Add internal _borrowed() functions that return Cow<\'a, str> to prepare for
zero-copy tokenization. When the source string needs no transformation
(no escaping), return Cow::Borrowed. When transformation is required,
return Cow::Owned.
The Token enum still uses String, so borrowed values are converted via
to_owned() for now. This maintains API compatibility while preparing the
codebase for a future refactor where Token can hold borrowed strings.
Optimized: comments, quoted strings, dollar-quoted strings, quoted identifiers.
0 commit comments