Skip to content

Commit 2635435

Browse files
committed
fix: add pad token setting and clarify tokenization in finetuning notebook
1 parent 55614aa commit 2635435

1 file changed

Lines changed: 3 additions & 1 deletion

File tree

ai/Finetuning/finetune.ipynb

Lines changed: 3 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -114,6 +114,8 @@
114114
"tokenizer = \n",
115115
"model =\n",
116116
"\n",
117+
"# Set pad token (because the end of the sentence is not detected by the model)\n",
118+
"tokenizer.pad_token =\n",
117119
"\n",
118120
"print(f\"✅ Model '{model_name}' loaded successfully!\")\n",
119121
"print(f\"Model has {model.num_parameters():,} parameters\")\n"
@@ -237,7 +239,7 @@
237239
"\n",
238240
"# 2. Tokeniser nos données (transformer le texte en nombres)\n",
239241
"def tokenize_function(examples):\n",
240-
" format_function(examples)\n",
242+
" texts = format_function(examples)\n",
241243
" \n",
242244
" # On n'utilise PAS return_tensors ici car Dataset.map() attend des listes, pas des tensors\n",
243245
" tokenized = tokenizer(\n",

0 commit comments

Comments
 (0)