Bug: error raised when trying to consume the eos token in final state

[RobinPicard]

Describe the issue as clearly as possible:

When the guide's current state is the final state, an error is raised when calling guide.advance with the eos token as an argument. This stands in contradiction to the transitions map that always indicates that the eos token can be consumed when in the final state (it leads to the same current state).

This is a serious problem when doing batch generation with a regex that contains several final states (for instance [1][{1,3}). In this case, if one of the generations reaches a an early final state, the logits processor will stop advancing through the guide (to avoid the bug), but that means that the model may keep generating tokens that would lead to a later final state! Since the current state is not updated, it will generate a string typically too long that does not respect the regex.

Steps/code to reproduce the bug:

import outlines
import transformers
from outlines.types import Regex
from outlines_core import Guide

model = outlines.from_transformers(
    transformers.AutoModelForCausalLM.from_pretrained("erwanf/gpt2-mini"),
    transformers.AutoTokenizer.from_pretrained("erwanf/gpt2-mini"),
)

generator = outlines.Generator(model, Regex(r"[1]{1,3}"))
index = generator.logits_processor.index
print(index)
guide = Guide(index)

for i in range(3):
    guide.advance(token_id=16, return_tokens=False)
guide.advance(token_id=50256, return_tokens=False)

Expected result:

No error raised

Error message:

Index object with transitions:
16 -> {
    50256: 16,
}
12 -> {
    16: 16,
    50256: 12,
}
28 -> {
    1157: 16,
    16: 12,
    50256: 28,
}
24 -> {
    1157: 12,
    16243: 16,
    16: 28,
}

Traceback (most recent call last):
  File "/home/robinpicard/outlines/.idea/dsl.py", line 34, in <module>
    guide.advance(token_id=50256, return_tokens=False)
ValueError: No next state found for the current state: 16 with token ID: 50256

Outlines/Python version information:

outlines-core 0.2.11
python 3.12

Context for the issue:

This makes the implementation of outlines-core in outlines buggy.

[agourdel]

In order to fix this issue,
just remove lines 229;230;231 from src/index.rs inside the "next_state" function and it should do the trick.

When we are in one final state, the eos_token will lead to the same final state as it's already implemented in the index constructor :

[RobinPicard]

Yes I wanted to do that initially, but I'm worried it would affect some users that expect the eos_token not be accepted in their current implementation. For example, there's the following in vllm:

    def accept_tokens(self, request_id: str, tokens: list[int]) -> bool:
        """Accepts a list of tokens and advances the FSM.

        Returns True if the FSM was advanced successfully.
        Returns False if the FSM failed to advance.
        """
        if self.guide.accepts_tokens(tokens):
            # Advance cannot fail because we checked Guide.accepts_tokens()
            for t in tokens:
                self.guide.advance(t)
                self.num_processed_tokens += 1
            return True
        return False

With this change True would be returned instead of False and num_processed_tokens would get incremented in some situations.

[agourdel]

Well, I saw what you proposed in #236 and I'm afraid it would be a mistake.

You need to keep the eos_token in the allowed tokens mask because otherwise the inference process will never know when to stop.
(eos should be one of the possibilites or the only one in the mask in order to let the model close the inference loop).

I understand the concern :
if you have one final state which is also a transition state , you can have a sequence of tokens where EOS_TOKEN is in the middle of the list (not at the end) and the list is still "valid" from a pure DFA's point of view.

But the fact that the EOS_TOKEN is a "special" token is a business rules outside the guide.
So from software architecture's point of view, this is the responsability of the user to know he has to deal differently with.
But from the Guide's point of view you have to let the user know that "here", an eos_token is possible.

[RobinPicard]

Yes, I closed the PR a few minutes ago as I also realized that it's not a viable option.