[SPARK-54099][SQL] XML variant parser should fall back to string on decimal parsing errors #52801
+33
−12
Conversation
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
sandip-db
suggested changes
Oct 31, 2025
sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/xml/StaxXmlParser.scala
Outdated
Show resolved
Hide resolved
sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/xml/StaxXmlParser.scala
Outdated
Show resolved
Hide resolved
sandip-db
approved these changes
Oct 31, 2025
sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/xml/StaxXmlParser.scala
Outdated
Show resolved
Hide resolved
| } | ||
|
|
||
| // Try parsing the value as decimal | ||
| val decimalParser = ExprUtils.getDecimalParser(options.locale) |
There was a problem hiding this comment.
Unrelated to the issue at hand, decimalParser should be reused rather than initializing for every value.
Can the caller pass it instead?
There was a problem hiding this comment.
We may have had this conversation in the original PR. Yes, we can, but we need to change a few function signatures. We can improve it later.
|
Merged to master and branch-4.1. |
HyukjinKwon
pushed a commit
that referenced
this pull request
Nov 2, 2025
…ecimal parsing errors
### What changes were proposed in this pull request?
When parsing XML data with `parse_xml` that contains decimal numbers with very large
exponents (e.g., "1E+2147483647"), the conversion to Variant type fails with:
```
java.lang.ArithmeticException: BigInteger would overflow supported range
at java.base/java.math.BigDecimal.setScale(BigDecimal.java:3000)
at org.apache.spark.sql.catalyst.xml.StaxXmlParser$.org$apache$spark$sql$catalyst$xml$StaxXmlParser$$appendXMLCharacterToVariant(StaxXmlParser.scala:1335)
```
It's because the parser calls `setScale(0)` to normalize the decimal. When the scale is extremely negative (e.g., -2147483647), `setScale(0)` attempts to
multiply the unscaled value by 10^2147483647, causing BigInteger overflow.
This PR will catch all errors when parsing strings as decimal in the XML variant parser and fall back to string.
### Why are the changes needed?
Bug fix.
### Does this PR introduce _any_ user-facing change?
No.
### How was this patch tested?
New UT.
### Was this patch authored or co-authored using generative AI tooling?
No.
Closes #52801 from xiaonanyang-db/SPARK-54099.
Authored-by: Xiaonan Yang <xiaonan.yang@databricks.com>
Signed-off-by: Hyukjin Kwon <gurwls223@apache.org>
(cherry picked from commit 88eef06)
Signed-off-by: Hyukjin Kwon <gurwls223@apache.org>
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
3 participants
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
What changes were proposed in this pull request?
When parsing XML data with
parse_xmlthat contains decimal numbers with very largeexponents (e.g., "1E+2147483647"), the conversion to Variant type fails with:
It's because the parser calls
setScale(0)to normalize the decimal. When the scale is extremely negative (e.g., -2147483647),setScale(0)attempts tomultiply the unscaled value by 10^2147483647, causing BigInteger overflow.
This PR will catch all errors when parsing strings as decimal in the XML variant parser and fall back to string.
Why are the changes needed?
Bug fix.
Does this PR introduce any user-facing change?
No.
How was this patch tested?
New UT.
Was this patch authored or co-authored using generative AI tooling?
No.