Polars: returning dict from `map_elements` function - key order does not match return_dtype declaration?
19:35 27 Jul 2023

With this dtype defined (note reverse-alpha field order):

Similarity = pl.Struct([
    pl.Field('zavg', pl.Float64),
    pl.Field('ystd', pl.Float64),
    pl.Field('xbase_indices', pl.List(pl.Int64))
])

When I run this map_elements returning a fixed dict that matches Similarity in keys, type, and order:

df = pl.DataFrame({'x': [0, 1]})
df.with_columns((
    pl.col('x')
    .map_elements(lambda d: {'zavg':1.1, 'ystd': 0.1, 'xbase_indices': [2]},
           return_dtype=Similarity)
    .alias(col+'_Blah')
    for col in ['item_1A', 'item_7']))

I get this SchemaError with key order mismatch (note key order): SchemaError: expected output type

Struct([
  Field { name: "zavg", dtype: Float64 }, 
  Field { name: "ystd", dtype: Float64 }, 
  Field { name: "xbase_indices", dtype: List(Int64) }])

got (note alphabetical key order)

Struct([
  Field { name: "xbase_indices", dtype: List(Int64) }, 
  Field { name: "ystd", dtype: Float64 }, 
  Field { name: "zavg", dtype: Float64 }])

set return_dtype to the proper datatype

If I remove the return_dtype it works. Further experiments suggest Polars is reading the dict keys in alphabetical order.

Expected? How should I handle this?

python dataframe python-polars