feat: add support for pyspark equivalent explode_outer, posexplode/posexplode_outer (via explode_with_index)

[bcallender]

TL;DR

Add outer explode support that preserves null/empty arrays, plus position-aware variants. No breaking changes; default explode behavior unchanged.

What’s included

• Explode “outer” semantics:
  • DataFrame APIs: explode_outer(col), posexplode_outer(col)
  • New keep_null_and_empty: bool on Explode and ExplodeWithIndex logical plans
  • Physical exec honors keep_null_and_empty:
    • Preserves rows with empty/null arrays
    • Emits null for value; and for position in pos-explode variants
• Position-aware explode:
  • explode_with_index(col, index_name="index", value_name=None, keep_null_and_empty=False)
  • PySpark alias: posexplode(col) and posexplode_outer(col)
• Serde + Protobuf:
  • New ExplodeWithIndex message
  • Explode and ExplodeWithIndex carry keep_null_and_empty
  • Optional value_name handled correctly (None when omitted)
• Tests:
  • Outer behavior for explode/posexplode
  • Optional naming coverage (only value_name; only index_name; both; expression input)
  • Plan serde round-trip for ExplodeWithIndex (default/custom names, outer)

Behavior details

• Regular explode: filters null/empty arrays (unchanged)
• Outer explode: preserves all rows, emits nulls for missing elements
• Position column:
  • Non-outer: 0-based indices per original row
  • Outer: null position for empty/null arrays

Compatibility

• PySpark semantics for posexplode/posexplode_outer (0-based indices here; position nulls for outer)
• Backward compatible defaults
• Protobuf/serde extended; existing fields untouched

Examples (with results)

df = session.create_dataframe({
    "id": [1, 2, 3],
    "tags": [["red", "blue"], [], None],
})

• Regular explode (filters empty/null)

df.explode("tags").to_polars()

id	tags
1	red
1	blue

• Outer explode (preserves rows, yields nulls)

df.explode_outer("tags").to_polars()

id	tags
1	red
1	blue
2	null
3	null

• posexplode (position + value; filters empty/null)

df.posexplode("tags").to_polars()

id	pos	col
1	0	red
1	1	blue

• posexplode_outer (position + value; preserves rows; nulls for empty/null)

df.posexplode_outer("tags").to_polars()

id	pos	col
1	0	red
1	1	blue
2	null	null
3	null	null

• explode_with_index (default names; filters empty/null)

df2 = session.create_dataframe({
    "id": [1, 2],
    "tags": [["x", "y"], ["z"]],
})
df2.explode_with_index("tags").to_polars()

id	index	tags
1	0	x
1	1	y
2	0	z

• explode_with_index (custom names; outer behavior)

df3 = session.create_dataframe({
    "id": [1, 2, 3],
    "letters": [["a", "b"], [], None],
})
df3.explode_with_index("letters", index_name="pos", value_name="val", keep_null_and_empty=True).to_polars()

id	pos	val
1	0	a
1	1	b
2	null	null
3	null	null

Testing and quality

• Lints clean
• Full suite passing (1198 passed, 23 skipped)
• Added serde and API tests covering optional parameters and outer variants

[bcallender]

Warning

This pull request is not mergeable via GitHub because a downstack PR is open. Once all requirements are satisfied, merge this PR as a stack on Graphite.
Learn more

• feat: add support for pyspark equivalent explode_outer, posexplode/posexplode_outer (via explode_with_index) #249 👈 (View in Graphite)
• feat: implement regexp_* text functions #247
• feat: add array_* functions #243
• refactor: reorganize array expressions in proto #245
• feat: add support for sum_distinct, count_distinct, approx_count_distinct #242
• main

This stack of pull requests is managed by Graphite. Learn more about stacking.

[YoungVor]

Nearly finished a pass. Implementation and tests looks great, had a few comments on the API. I'll finish my pass tomorrow

[YoungVor]

maybe posexplode_with_fields?

[bcallender]

Hmm I don't think so -- this is just a superset of the functionality of those two functions, so it makes sense to keep it named separately, at least imo.

[YoungVor]

any reason we couldn't keep everything as 'pos', so pos_name, and default to 'pos'? Its slightly strange to me to have the descriptions and default depart from the posexplode functions

in the example you could show that you can change it to index.

[bcallender]

I still think it makes sense for the names of this function to be internally consistent (index), but I think it does make sense to change the default to pos to match the existing functions.

[YoungVor]

could call this 'outer', or at least mention that behavior would mimick explode/explode_outer

[bcallender]

added clarification. I think this is a more explicit name, but mentioned that it is the same as regular posexplode (False) or posexplode_outer (True).

[YoungVor]

Finished my pass, looks great! I love the consolidation of code with a single explode_with_index logical expression.

A few nits, but mostly around the API documentation