Unsupported tokenizer for LiquidAI/LFM2-1.2B

[HungNguyenPSL]

Describe the issue as clearly as possible:

Hi, I have an issue with LiquidAI/LFM2-1.2B model, which is not currently supporrted by outlines-core:

Steps/code to reproduce the bug:

from outlines_core import Index, Vocabulary

vocab = Vocabulary.from_pretrained("LiquidAI/LFM2-1.2B")

Expected result:

No error expected

Error message:

---------------------------------------------------------------------------
ValueError                                Traceback (most recent call last)
[/tmp/ipython-input-1035479692.py](https://localhost:8080/#) in <cell line: 0>()
      1 from outlines_core import Index, Vocabulary
      2 
----> 3 vocab = Vocabulary.from_pretrained("LiquidAI/LFM2-1.2B")

ValueError: Unsupported tokenizer for LiquidAI/LFM2-1.2B: Token processor, please open an issue with the full error message: https://github.com/dottxt-ai/outlines-core/issues

Outlines/Python version information:

Version information

Details

``` Python 3.12.12

Name: outlines_core
Version: 0.2.13

</details>


### Context for the issue:

I tried to run some experiments with Small Models and constrained decoding, but outlines-core does not support LiquidAI models.