fix(backends): handle single-digit month/day in pyspark to_date

[bhargaviHQ]

Description of changes

Spark's TO_DATE uses Java SimpleDateFormat where MM/dd require leading zeros, so dates like 1/1/2026 with format %m/%d/%Y return NaT.

Remapping %m/%d to %-m/%-d before sqlglot's format conversion produces Spark's M/d, which accepts both 1 and 2 digit values.

Issues closed

Resolves #12004

[NickCrews]

@bhargaviHQ Does this belong upstream, in sqlglot? If I remember right I thought they DID do conversion between "magic" strings, such as regexes, format strings, parsing strings, etc

[NickCrews]

I would think that the desired behavior for this test would be for it to pass on all backends, not just pyspark. Do you agree? Does it actually pass or do we need this pytest.mark.pyspark? I don't really like how we are only testing for pyspark behavior here. I want these backend-agnostic tests to be verifying that all backends perform the same, with a few pytest.mark.notimpl for the backends that we can't adapt.

[deepyaman]

@bhargaviHQ Does this belong upstream, in sqlglot? If I remember right I thought they DID do conversion between "magic" strings, such as regexes, format strings, parsing strings, etc

I agree that this probably belongs upstream. Here's an attempt I just pushed up: tobymao/sqlglot#7739

It could also be fixed in Ibis for now, but I think it should override the translation to output the correct SQL, rather than trying to hack the format string like this. (The only reference I see doing something similar in Ibis is the SingleStore backend, and I'm not extremely confident about that being the right approach...)

[bhargaviHQ]

Thanks for the feedback. @NickCrews - regarding the test, noted. I have updated it to be backend agnostic instead of pyspark only.

Regarding the fix itself @NickCrews and @deepyaman , agreed this is best fixed upstream. For now I have kept the remap as the interim fix, happy to make the required follow-up change in ibis once the sqlglot#7739 fix lands.

[deepyaman]

Thanks for the feedback. @NickCrews - regarding the test, noted. I have updated it to be backend agnostic instead of pyspark only.

Regarding the fix itself @NickCrews and @deepyaman , agreed this is best fixed upstream. For now I have kept the remap as the interim fix, happy to make the required follow-up change in ibis once the sqlglot#7739 fix lands.

I don't think there is a bug. 🤦

After trying to fix it upstream, and doing some more investigation, I find I can't actually reproduce the reported issue. @bhargaviHQ I undid your "fix", and all the PySpark tests still pass.

I think there may be something that doesn't work, but the specific issue that was reported doesn't seem to be something either of us have properly replicated.

[deepyaman]

Thanks for the feedback. @NickCrews - regarding the test, noted. I have updated it to be backend agnostic instead of pyspark only.
Regarding the fix itself @NickCrews and @deepyaman , agreed this is best fixed upstream. For now I have kept the remap as the interim fix, happy to make the required follow-up change in ibis once the sqlglot#7739 fix lands.

I don't think there is a bug. 🤦

After trying to fix it upstream, and doing some more investigation, I find I can't actually reproduce the reported issue. @bhargaviHQ I undid your "fix", and all the PySpark tests still pass.

I think there may be something that doesn't work, but the specific issue that was reported doesn't seem to be something either of us have properly replicated.

Actually, I'm wrong, there probably is a bug if you aren't specifying legacy mode; see #12004 (comment).

[deepyaman]

@bhargaviHQ I've helped fix this in SQLGlot upstream (see tobymao/sqlglot#7773), and opened #12021 that takes that fix into account, so I'm closing this.