[Variant] Enforce shredded-type validation in `shred_variant` #8796

liamzwbao · 2025-11-05T23:45:26Z

Which issue does this PR close?

Closes [Variant] Enforce shredded-type validation in shred_variant #8795.

Rationale for this change

Mentioned in the issue

What changes are included in this PR?

Add validation in shred_variant to allow spec-approved types only.

Are these changes tested?

Yes

Are there any user-facing changes?

liamzwbao

Hi @klion26, this is the PR to address the confusion in #8768 (comment).

There’s some noise from fmt, please review with “Hide whitespace” on.

cc @alamb @scovich

liamzwbao · 2025-11-06T01:02:35Z

parquet-variant-compute/src/variant_to_arrow.rs

-        DataType::FixedSizeBinary(size) => {
-            return Err(ArrowError::InvalidArgumentError(format!(
-                "FixedSizeBinary({size}) is not a valid variant shredding type. Only FixedSizeBinary(16) for UUID is supported."
-            )));
-        }


Moved to shred_variant

liamzwbao · 2025-11-06T01:19:05Z

parquet-variant-compute/src/variant_to_arrow.rs

-        _ if data_type.is_primitive() => {
-            return Err(ArrowError::NotYetImplemented(format!(
-                "Primitive data_type {data_type:?} not yet implemented"
-            )));
-        }
-        _ => {
-            return Err(ArrowError::InvalidArgumentError(format!(
-                "Not a primitive type: {data_type:?}"


Here, primitive refers to Variant primitive, not Arrow primitive. So we shouldn’t throw an error about “not an Arrow primitive.” Ideally this function should accept any Arrow type, therefore changed this to NotYetImplemented instead.

klion26 · 2025-11-06T10:46:20Z

parquet-variant-compute/src/shred_variant.rs

-        _ => {
+        // Supported shredded primitive types, see Variant shredding spec:
+        // https://github.com/apache/parquet-format/blob/master/VariantShredding.md#shredded-value-types
+        DataType::Boolean


We can't make do the type cast after this? not sure if this is ok.

Currently, we will do the type check in make_primitive_variant_to_arrow_row_builder with the match arms, so maybe we don't need to add it here, and add the type check in two places seems will add maintaince.

The issue is that make_primitive_variant_to_arrow_row_builder doesn’t enforce the Parquet-primitive constraint. As a result, types like UInt, which aren’t Parquet primitives, are accepted for shredding because make_primitive_variant_to_arrow_row_builder allows.

klion26 · 2025-11-06T10:53:28Z

parquet-variant-compute/src/shred_variant.rs

            VariantToShreddedVariantRowBuilder::Primitive(typed_value_builder)
        }
+        DataType::FixedSizeBinary(_) => {
+            return Err(ArrowError::InvalidArgumentError(format!("{data_type} is not a valid variant shredding type. Only FixedSizeBinary(16) for UUID is supported.")))


Do we need to distinguish this with the _ match arm?

I think this provides a user friendly message and it's actually moved from variant_to_arrow. Don't mind removing it tho

klion26 · 2025-11-06T11:16:41Z

parquet-variant-compute/src/variant_to_arrow.rs

+                StringView(VariantToStringArrowBuilder::new(cast_options, capacity))
+            }
+            _ => {
+                return Err(ArrowError::NotYetImplemented(format!(


Maybe we can't remove the _ if data_type.is_primitive() for now, seems this arm is for the type we may but have not implement, and _ match arm is for the types invalid.

After #8768 merged, we complete the 1-1 mapping(and some transforms for some types) for all Variant primitive types, but we may support some DataTypes here which is not a valid variant primitive(e.g. Timestamp with different unit), and keep the _ if data_type.is_primitive() so that we know we may support the required, and remove it after we have a conclusion.

I am sorting out possible conversions and will create an issue to discuss them after the work done.

Actually data_type.is_primitive() is not checking Parquet primitive, that's why I wanted to remove it to avoid confusion. It checks arrow primitive which only includes numeric and temporal. If we were to keep it, we shouldn't add Binary or String into this builder as they are not arrow primitive.

My understanding is that

All Parquet Variant primitive types will map to Arrow DataTypes

Some Parquet Variant primitive types can be converted to other Arrow DataTypes(with cast)

Here we want to support all parquet variant primitive types, and the target type here is Arrow DataType, we check the Arrow DataTypes here

The dataype.is_primitive() check(my guess) is that we can support all Arrow DataTypes, but has not done(when this code written), so add a NotYetImplement here to mark this.

However, it is true that we might need an expert to confirm our understanding here.

All Parquet Variant primitive types will map to Arrow DataTypes

Some Parquet Variant primitive types can be converted to other Arrow DataTypes(with cast)

Here we want to support all parquet variant primitive types, and the target type here is Arrow DataType, we check the Arrow DataTypes here

That's my understanding as well.

The dataype.is_primitive() check(my guess) is that we can support all Arrow DataTypes, but has not done(when this code written), so add a NotYetImplement here to mark this.

This is why I’m proposing the change. I’m not removing NotYetImplemented. Instead, I’m removing the datatype.is_primitive() check and the InvalidArgumentError: Not a primitive type, because, as you noted, we can support all Arrow DataTypes.

Example: variant_get() with Binary

Before this change: fails with InvalidArgumentError: Not a primitive type because Binary is not an Arrow primitive. This incorrectly suggests Binary is an invalid target type and shouldn’t be supported.

After this change: fails with NotYetImplemented, which is the expected behavior until Binary is supported.

Hope this example makes sense

scovich

Generally looks good, but trips over UUID extension type issues. I'm not sure if we should merge as-is with follow-up items to fix those once #8420 is addressed? Or wait for #8673 to merge so we have access to the field and not just a bare data type?

scovich · 2025-11-07T15:08:52Z

parquet-variant-compute/src/shred_variant.rs

+        // https://github.com/apache/parquet-format/blob/master/VariantShredding.md#shredded-value-types
+        DataType::Boolean


Big question:

Suggested change

// https://github.com/apache/parquet-format/blob/master/VariantShredding.md#shredded-value-types

DataType::Boolean

// https://github.com/apache/parquet-format/blob/master/VariantShredding.md#shredded-value-types

DataType::Null

| DataType::Boolean

I'm pretty sure that Variant::Null (JSON null) is an actual value... but I'm NOT sure how useful it would be in practice. Strict casting would produce an error if even one row had a normal value, and non-strict casting would produce an all-null output no matter what the input was.

So maybe we intentionally forbid DataType::Null, with an explanatory comment?

Update:

I confused myself; the variant shredding spec does not allow shredding as null, so this code is correct as-is.

HOWEVER, while looking at this I realized that we do have a row builder for DataType::Null, and it does not currently enforce strict casting (all values are blindly treated as null, no matter the casting mode).

The simplest fix would be to adjust the null builder's definition:

define_variant_to_primitive_builder!( struct VariantToNullArrowRowBuilder<'a> |capacity| -> FakeNullBuilder { FakeNullBuilder::new(capacity) }, - |_value| Some(Variant::Null), + |value| value.as_null(), type_name: "Null" );

and change the fake row builder's append_value method to suit:

impl FakeNullBuilder { fn new(capacity: usize) -> Self { Self(NullArray::new(capacity)) } - fn append_value<T>(&mut self, _: T) {} + fn append_value(&mut self, _: ()) {} fn append_null(&mut self) {}

... but that might produce clippy warnings about passing unit type as a function argument. If so, we'd need to adjust the value conversion to produce Some dummy value instead, e.g. value.as_null().map(|_| 0) or matches!(value, Variant::Null).then_some(0)

Also, the fake null builder should probably track how many "values" were "appended" and either produce a NullArray of that length or blow up if the call count disagrees with the array's length. The former is probably more correct than the latter, since it matches all the other builders for whom "capacity" is only a hint.

The latter is unrelated to this PR, tracking it as #8810

scovich · 2025-11-07T15:13:10Z

parquet-variant-compute/src/shred_variant.rs

+        DataType::FixedSizeBinary(_) => {
+            return Err(ArrowError::InvalidArgumentError(format!("{data_type} is not a valid variant shredding type. Only FixedSizeBinary(16) for UUID is supported.")))
+        }


Not sure whether this distinction is important enough to merit its own error message?

Also, we eventually need to check the field for UUID extension type and not just rely on the data type. If #8673 merges first, we should fix it here; if this PR merges first the other PR needs to incorporate the change.

CC @friendlymatthew

scovich · 2025-11-07T15:34:39Z

parquet-variant-compute/src/variant_to_arrow.rs

+            _ => {
+                return Err(ArrowError::NotYetImplemented(format!(
+                    "Data_type {data_type:?} not yet implemented"


Instead of a catch-all, should we enumerate the remaining data types so it's obvious at a glance which ones are still unsupported? I don't think there are very many left.

Makes sense to me, done

scovich · 2025-11-07T15:35:20Z

parquet-variant-compute/src/variant_to_arrow.rs

-    let builder = match data_type {
-        DataType::Null => Null(VariantToNullArrowRowBuilder::new(cast_options, capacity)),
-        DataType::Boolean => Boolean(VariantToBooleanArrowRowBuilder::new(cast_options, capacity)),
-        DataType::Int8 => Int8(VariantToPrimitiveArrowRowBuilder::new(
-            cast_options,
-            capacity,
-        )),
-        DataType::Int16 => Int16(VariantToPrimitiveArrowRowBuilder::new(
-            cast_options,
-            capacity,
-        )),
-        DataType::Int32 => Int32(VariantToPrimitiveArrowRowBuilder::new(
-            cast_options,
-            capacity,
-        )),
-        DataType::Int64 => Int64(VariantToPrimitiveArrowRowBuilder::new(
-            cast_options,
-            capacity,
-        )),
-        DataType::UInt8 => UInt8(VariantToPrimitiveArrowRowBuilder::new(
-            cast_options,
-            capacity,
-        )),
-        DataType::UInt16 => UInt16(VariantToPrimitiveArrowRowBuilder::new(
-            cast_options,
-            capacity,
-        )),
-        DataType::UInt32 => UInt32(VariantToPrimitiveArrowRowBuilder::new(
-            cast_options,
-            capacity,
-        )),
-        DataType::UInt64 => UInt64(VariantToPrimitiveArrowRowBuilder::new(
-            cast_options,
-            capacity,
-        )),
-        DataType::Float16 => Float16(VariantToPrimitiveArrowRowBuilder::new(
-            cast_options,
-            capacity,
-        )),
-        DataType::Float32 => Float32(VariantToPrimitiveArrowRowBuilder::new(
-            cast_options,
-            capacity,
-        )),
-        DataType::Float64 => Float64(VariantToPrimitiveArrowRowBuilder::new(
-            cast_options,
-            capacity,
-        )),
-        DataType::Decimal32(precision, scale) => Decimal32(VariantToDecimalArrowRowBuilder::new(
-            cast_options,
-            capacity,
-            *precision,
-            *scale,
-        )?),
-        DataType::Decimal64(precision, scale) => Decimal64(VariantToDecimalArrowRowBuilder::new(
-            cast_options,
-            capacity,
-            *precision,
-            *scale,
-        )?),
-        DataType::Decimal128(precision, scale) => Decimal128(VariantToDecimalArrowRowBuilder::new(
-            cast_options,
-            capacity,
-            *precision,
-            *scale,
-        )?),
-        DataType::Decimal256(precision, scale) => Decimal256(VariantToDecimalArrowRowBuilder::new(
-            cast_options,
-            capacity,
-            *precision,
-            *scale,
-        )?),
-        DataType::Timestamp(TimeUnit::Microsecond, None) => TimestampMicroNtz(
-            VariantToTimestampNtzArrowRowBuilder::new(cast_options, capacity),
-        ),
-        DataType::Timestamp(TimeUnit::Microsecond, tz) => TimestampMicro(
-            VariantToTimestampArrowRowBuilder::new(cast_options, capacity, tz.clone()),
-        ),
-        DataType::Timestamp(TimeUnit::Nanosecond, None) => TimestampNanoNtz(
-            VariantToTimestampNtzArrowRowBuilder::new(cast_options, capacity),
-        ),
-        DataType::Timestamp(TimeUnit::Nanosecond, tz) => TimestampNano(
-            VariantToTimestampArrowRowBuilder::new(cast_options, capacity, tz.clone()),
-        ),
-        DataType::Date32 => Date(VariantToPrimitiveArrowRowBuilder::new(
-            cast_options,
-            capacity,
-        )),
-        DataType::Time64(TimeUnit::Microsecond) => Time(VariantToPrimitiveArrowRowBuilder::new(
-            cast_options,
-            capacity,
-        )),
-        DataType::FixedSizeBinary(16) => {
-            Uuid(VariantToUuidArrowRowBuilder::new(cast_options, capacity))
-        }
-        DataType::FixedSizeBinary(size) => {
-            return Err(ArrowError::InvalidArgumentError(format!(
-                "FixedSizeBinary({size}) is not a valid variant shredding type. Only FixedSizeBinary(16) for UUID is supported."
-            )));
-        }
-        DataType::Utf8 => String(VariantToStringArrowBuilder::new(cast_options, capacity)),
-        DataType::LargeUtf8 => {
-            LargeString(VariantToStringArrowBuilder::new(cast_options, capacity))
-        }
-        DataType::Utf8View => StringView(VariantToStringArrowBuilder::new(cast_options, capacity)),
-        _ if data_type.is_primitive() => {
-            return Err(ArrowError::NotYetImplemented(format!(
-                "Primitive data_type {data_type:?} not yet implemented"
-            )));
-        }
-        _ => {
-            return Err(ArrowError::InvalidArgumentError(format!(
-                "Not a primitive type: {data_type:?}"
-            )));
-        }
-    };
+    let builder =
+        match data_type {


aside: Thanks, clippy... (best to review with whitespace ignored)

scovich · 2025-11-07T15:39:19Z

parquet-variant-compute/src/variant_to_arrow.rs

+            DataType::FixedSizeBinary(16) => {
+                Uuid(VariantToUuidArrowRowBuilder::new(cast_options, capacity))
+            }


This is actually incorrect. It's perfectly legal to (attempt to) convert any Variant::Binary (plus maybe also Variant::String and Variant::Uuid) value to fixed-len binary -- including length 16. The conversion just might fail if the length is wrong.

We really need a UUID extension type check to know whether the caller intended to work with UUID or binary values.

I agree, which is why I removed the branch that errored on FixedSizeBinary(size). But this issue should be handled in a separate PR as it needs to change the impl the FixedSizeBinary. I'll add a TODO here.

klion26

Thanks for the update, LGTM.

alamb

This makes sense to me -- thank you @liamzwbao and @klion26

[Variant] Enforce shredded-type validation in shred_variant

98c2469

github-actions bot added the parquet-variant parquet-variant* crates label Nov 5, 2025

fmt and doc

9ecdfca

liamzwbao commented Nov 6, 2025

View reviewed changes

liamzwbao marked this pull request as ready for review November 6, 2025 01:15

liamzwbao commented Nov 6, 2025

View reviewed changes

klion26 reviewed Nov 6, 2025

View reviewed changes

alamb mentioned this pull request Nov 6, 2025

Andrew Lamb Weekly-ish Open Source plan - 2025-11-03 apache/datafusion#18486

Open

52 tasks

scovich reviewed Nov 7, 2025

View reviewed changes

scovich mentioned this pull request Nov 7, 2025

Variant to NullType conversion ignores strict casting #8810

Open

liamzwbao added 3 commits November 10, 2025 18:09

Merge branch 'main' into issue-8795-shred-variant-validation

c293b80

Explicitly list all datatypes

aca092c

Add tests

a6e38ce

klion26 approved these changes Nov 12, 2025

View reviewed changes

alamb approved these changes Nov 13, 2025

View reviewed changes

		// https://github.com/apache/parquet-format/blob/master/VariantShredding.md#shredded-value-types
		DataType::Boolean

[Variant] Enforce shredded-type validation in shred_variant #8796

Are you sure you want to change the base?

[Variant] Enforce shredded-type validation in shred_variant #8796

Uh oh!

Conversation

liamzwbao commented Nov 5, 2025

Which issue does this PR close?

Rationale for this change

What changes are included in this PR?

Are these changes tested?

Are there any user-facing changes?

Uh oh!

liamzwbao left a comment

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

liamzwbao Nov 6, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

liamzwbao Nov 6, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

scovich left a comment

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

liamzwbao Nov 10, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

klion26 left a comment

Choose a reason for hiding this comment

Uh oh!

alamb left a comment

Choose a reason for hiding this comment

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

4 participants

[Variant] Enforce shredded-type validation in `shred_variant` #8796

[Variant] Enforce shredded-type validation in `shred_variant` #8796

liamzwbao Nov 6, 2025 •

edited

Loading

liamzwbao Nov 6, 2025 •

edited

Loading

liamzwbao Nov 10, 2025 •

edited

Loading