Syllable-aware BPE tokenizer for the Amharic language (አማርኛ) – fast, accurate, trainable.

A ridiculously fast Python BPE (Byte Pair Encoder) implementation written in Rust

A PHP implementation of OpenAI's BPE tokenizer tiktoken.

Byte-Pair Encoding tokenizer for training large language models on huge datasets

Tok: my own Tokenizer

High-Performance Tokenizer implementation in PHP.

BPE tokenizer for LLMs in Pure Zig

Teaching transformer-based architectures

(1) Train large language models to help people with automatic essay scoring. (2) Extract essay features and train new tokenizer to build tree models for score prediction.

implementation of Byte-Pair Encoding (BPE) for subword tokenization, written entirely in C++ . The tokenizer learns merges from raw text and supports encoding/decoding with UTF-8

🐍This is a fast, lightweight, and clean CPython extension for the Byte Pair Encoding (BPE) algorithm, which is commonly used in LLM tokenization and NLP tasks.

a parallel and minimal implementation of Byte Pair Encoding (BPE) from scratch in less than 200 lines of python.

[Rust] Unofficial implementation of "SuperBPE: Space Travel for Language Models" in Rust

High performance Byte-Pair Encoding tokenizer for large language models

Build a light-weight Llama from scratch, based on course Stanford CS336 2025.

Transformer Models for Humorous Text Generation. Fine-tuned on Russian jokes dataset with ALiBi, RoPE, GQA, and SwiGLU.Plus a custom Byte-level BPE tokenizer.

This project implements a Byte Pair Encoding (BPE) tokenization approach along with a Word2Vec model to generate word embeddings from a text corpus. The implementation leverages Apache Hadoop for distributed processing and includes evaluation metrics for optimal dimensionality of embeddings.

Fast BPE tokenizer with guaranteed perfect text reconstruction. 2x faster than tiktoken at encoding/decoding while preserving all whitespace and formatting.

processing de LANguage NATural

C89, single header, nostdlib byte pair encoding algorythm