Search results

Reproducible WordPiece encode bench shows ~5x speedup on 123B fixture — methodology question for maintainers

Status: Open.

#2075 In huggingface/tokenizers;

· SMC17 opened on May 28, 2026

Unigram trainer: prune loss uses alternatives.len() instead of alternatives[id].len()

Status: Open.

#2069 In huggingface/tokenizers;

· hunter-heidenreich opened on May 24, 2026

BPE trainer: i32 overflow in pair_counts corrupts merge ordering on large code corpora

Status: Open.

#2058 In huggingface/tokenizers;

· petyb opened on May 14, 2026

Publish error in npm repository, Node.js binary bindings

Status: Open.

#2045 In huggingface/tokenizers;

· uasan opened on Apr 28, 2026

It is important to understand the real cause of the problem.

Status: Open.

#2027 In huggingface/tokenizers;

· Nearbe opened on Apr 18, 2026

Feature request: compact vocabulary remapping for domain-specific training with pretrained tokenizers

Status: Open.

#2023 In huggingface/tokenizers;

· saslifat-gif opened on Apr 14, 2026

[RFC] Korean Tokenization: Jamo Decomposition as Pre-tokenizer (A Blueprint for Compositional Scripts)

Feature Request

Status: Open.

#1975 In huggingface/tokenizers;

· nicezic opened on Mar 22, 2026

Make huggingface-hub an optional dependency (extras)

Feature Request

Status: Open.

#1973 In huggingface/tokenizers;

· clemlesne opened on Mar 20, 2026

Serialization of Bytelevel for decoder and pre_tokenizer is missleading

Status: Open.

#1960 In huggingface/tokenizers;

· ArthurZucker opened on Mar 4, 2026

Tokenizer encode ignores pre-tokenizer when using right to left (rtl) digit splits with add_tokens

Status: Open.

#1957 In huggingface/tokenizers;

· gsaltintas opened on Mar 1, 2026

Constructive criticism for Tokenizers library.

Status: Open.

#1954 In huggingface/tokenizers;

· BBC-Esq opened on Feb 23, 2026

encode_batch has suboptimal parallelization on high-core systems

Status: Open.

#1929 In huggingface/tokenizers;

· stargazerZJ opened on Jan 14, 2026