WIP: working on byte-pair tokenizer;

basic overview done, decoding/encoding in progress;
44:00 @ yt video

Author: MKaczkow

README.md

@@ -46,4 +46,5 @@ Repo for learning tools and ideas of ML/AI
 * [pytorch profiler](./tools/pytorch_profiler/)
 * [kernels](./theory/kernels/)
 * [model context protocol](./tools/model_context_protocol/)
-* [grammar constrained decoding](./ideas/grammar_constrained_decoding/)
+* [grammar constrained decoding](./ideas/grammar_constrained_decoding/)
+* [byte-pair endcoding tokenier](./theory/bpe_tokenizer/)

pyproject.toml

@@ -8,6 +8,8 @@ dependencies = [
     "guidance>=0.2.3",
     "ipykernel>=6.29.5",
     "llama-cpp-python>=0.3.9",
+    "sentencepiece>=0.2.1",
+    "tiktoken>=0.12.0",
     "torch>=2.7.1",
 ]