Skip to content

fix(bpe): disambiguate legacy v1 merges with spaces in vocab tokens - #2078

Closed
SMC17 wants to merge 1 commit into
huggingface:mainfrom
SMC17:bpe-legacy-v1-merges-with-spaces
Closed

fix(bpe): disambiguate legacy v1 merges with spaces in vocab tokens#2078
SMC17 wants to merge 1 commit into
huggingface:mainfrom
SMC17:bpe-legacy-v1-merges-with-spaces

bpe: disambiguate legacy v1 string-merge tokens containing spaces

352b3e6
Select commit
Loading
Failed to load commit list.

Workflow runs completed with no jobs