fix: classify unsupported format patterns as Unsupported in CometFromUnixTime#4660
Open
marvelshan wants to merge 3 commits into
Open
fix: classify unsupported format patterns as Unsupported in CometFromUnixTime#4660marvelshan wants to merge 3 commits into
marvelshan wants to merge 3 commits into
Conversation
marvelshan
force-pushed
the
fix/from-unixtime-unsupported-format
branch
from
0aa73b7 to
32f11be
Compare
coderfender
reviewed
Jun 16, 2026
| val timeZone = exprToProtoInternal(Literal(expr.timeZoneId.orNull), inputs, binding) | ||
|
|
||
| if (expr.format != Literal(TimestampFormatter.defaultPattern)) { | ||
| if (expr.format != Literal(TimestampFormatter.defaultPattern())) { |
Contributor
There was a problem hiding this comment.
nit : might be an unwanted change ?
| if (expr.format != Literal(TimestampFormatter.defaultPattern())) { | ||
| Unsupported(Some("Only the default datetime pattern `yyyy-MM-dd HH:mm:ss` is supported")) | ||
| } else { | ||
| Incompatible(None) |
Contributor
There was a problem hiding this comment.
We might wnat to add some SQL tests to verify the fallback and make sure the plan is annotated
…UnixTime CometFromUnixTime previously reported all limitations as Incompatible, but non-default format patterns cannot be executed natively at all and should be Unsupported. The DataFusion timestamp range difference is the genuine Incompatible case. - Split getIncompatibleReasons to only contain the timestamp range issue - Add getUnsupportedReasons for the format pattern limitation - Make getSupportLevel return Unsupported for non-default patterns, Incompatible for the default pattern
…patterns Verify that non-default format patterns fall back to Spark with the expected reason, and that the default pattern runs natively when allowIncompatible=true.
marvelshan
force-pushed
the
fix/from-unixtime-unsupported-format
branch
from
32f11be to
737eb77
Compare
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Which issue does this PR close?
Closes #4575
Rationale for this change
CometFromUnixTimeincorrectly reports non-default datetime format patterns asIncompatibleinstead ofUnsupported. This is misleading because:Incompatibleimplies the expression can run natively if the user opts in viaspark.comet.expr.allowIncompatible=true, but non-default formats cannot be executed natively at all — theconvert()method returnsNonefor them regardless of configuration.GenerateDocstool usesgetIncompatibleReasons()andgetUnsupportedReasons()to produce the public Compatibility Guide. Classifying format limitations as "incompatible" misleads users into thinking they can enable support by settingallowIncompatible=true.What changes are included in this PR?
getIncompatibleReasons()to only contain the timestamp range difference (which is a true incompatibility — native can execute but results may differ at boundaries)getUnsupportedReasons()to document format pattern limitations (native cannot execute these at all)getSupportLevel(expr)to dynamically returnUnsupportedfor non-default format patterns andIncompatiblefor the default pattern, matching the actual behavior inconvert()How are these changes tested?
Existing tests cover the fallback behavior. The
convert()method already returnsNonefor non-default formats, andgetSupportLevel()now aligns with that behavior. TheGenerateDocsoutput will correctly classify format limitations under "Unsupported" rather than "Incompatible".