Feat req: make the characters strategy even more user friendly, for non-injective codecs and custom control characters.

[JamesParrott]

Hi everyone,
In PyShp, we've had informal legacy support for every single text codec in Python (91 of them), and I've only recently got around to adding tests for these. Hypothesis has been incredibly helpful for that, and helped both uncover some lurking bugs, and point the way to making our unicode handling robust. Huge thanks to the whole team.

There were just a few issues I had to add custom code for, that I think must also be applicable to any project that round trips strings to bytes via all the unicode codecs.

Firstly there are stateful codecs such as the ISO 2002 ones. Even though I added exclude_categories=["Cs", "Co", "Cn"], the characters strategy, with the codec "iso2022_jp" for example still presumably draws from Python's entire quick mapping which includes, the codec's custom (non-unicode control-char-categorised) control char: '\x1b' This can indeed be encoded, but decoding the result (b'\x1b') back again errors.

Secondly there are non-injective codecs. Under these two code points can encode to the same bytes. I don't know of any stochastic codecs (thankfully) so for now they can all only decode these bytes to one of them. I would like a strategy (or an option in characters) to omit the ones, that cannot be restored by decoded its encoded bytes back again. E.g under "cp950", "•" (U+2022) and "" (U+2027) both encode to b'\xa1E'

>>> enc = "cp950"
>>> "•".encode(enc) == "‧".encode(enc)
True

Is there any interest in adding some options, so that users can instruct hypothesis to avoid generating these troublesome characters, for these codecs? Currently I hardcode a few codecs, and assemble a custom list of exclude_characters along with each codec.

Thanks,

[Zac-HD]

Hi James - I'm glad to hear that Hypothesis has been helpful for you! Your current approach of calculating an exclude_characters set per-codec is exactly what I'd recommend given our current interface. If we were to add some further support for this use-case, there would be a couple of design questions to answer:

1. What should the new behavior be? I think the current setup, where we take an expansive view by default, is appropriate - because IMO it's usually better to confront users with an immediate test failure (and the tools to fix it, here exclude_characters) than to silently skip over part of the input space where bugs tend to hide.
2. If we added an option to this, what would the interface look like? st.characters() already takes quite a few arguments, and confusing new users or those who know less about unicode and codecs is a real cost to balance against the benefit.
   a. perhaps a helper to compute non-decodable and non-bijective characters given the name of codec, which can then be passed to exclude_characters if desired? Our caching would be nice for this too.
3. [and we have to implement it, but that's pretty easy given a design]

On brief consideration I'm inclined to write this up as a blog / how-to / recipe, rather than changing the Hypothesis interface per se, because I think it's neat but probably also fairly niche (although, so is explicit use of st.characters()). Thoughts?

[JamesParrott]

Hi Zac,

Thanks for the reply. A link from the docs to an article, especially with a recipe would work great too though. Documentation-only solutions are quite elegant (and LLM friendly for these modern times), and not many users will require any more than that.

Number 1) is a great point though - running into this error via Hypothesis did indeed cause me to consider how non-roundtrippable characters should be handled, in the library code as well, not just in the test code. It's definitely an opportunity to be more pro-actively helpful, and handle the edge case better.
Secondly, I take on board too, that the complexity of the API would increase, were this included in st.characters.
First of all, I wondered if the unicode "Co" Control Characters category (when present in exclude_categories) could be extended to include a specified codec's custom control characters. But this is a "nice to have", it's hardly a strong enough case to merit breaking compliance with Unicode spec.
After that, with my enthusiastic coder's hat on, I thought it would be simple to add a new boolean kwarg to st.characters, e.g. must_roundtrip=False. If it was True, and if a codec was specified, behind the scenes st.characters would loads a cache or analyse the codec when needed, and appends any non round-trippable code points found (for that codec) to exclude_characters. Perhaps the ramifications of that are not so simple though - not limited to the tests, documentation changes, and growing the API, that must be supported (unless an unusual breaking change shrank it).

I'm quite interested in this, but it is an edge case even for me. So admittedly I've not yet delved further, into what information about codecs is available directly from the Python core libraries. Nor have I considered Stateful Codecs much more than observing they rely on control characters.