Detect rows where uniqueness is not given in Polars
09:47 09 Mar 2023

Currently I have the following problem. I have to check if there is an infringement if the column values ID, table and value_a are not unique.

import polars as pl

df = pl.DataFrame(
    {
    "ID": ["1", "1", "1", "1", "1"],
    "column": ["foo", "foo", "bar", "ham", "egg"],
    "table": ["A", "A", "C", "D", "E"],
    "value_a": ["tree", tree, None, "bean", None,],
    "value_b": ["Lorem", "Ipsum", "Dal", "Curry", "Dish",],
    "mandatory": ["M", "M", "M", "CM", "M"]
    }
)
print(df)
shape: (5, 6)
┌─────┬────────┬───────┬─────────┬─────────┬───────────┐
│ ID  ┆ column ┆ table ┆ value_a ┆ value_b ┆ mandatory │
│ --- ┆ ---    ┆ ---   ┆ ---     ┆ ---     ┆ ---       │
│ str ┆ str    ┆ str   ┆ str     ┆ str     ┆ str       │
╞═════╪════════╪═══════╪═════════╪═════════╪═══════════╡
│ 1   ┆ foo    ┆ A     ┆ tree    ┆ Lorem   ┆ M         │
│ 1   ┆ foo    ┆ B     ┆ tree    ┆ Ipsum   ┆ M         │
│ 1   ┆ bar    ┆ C     ┆ null    ┆ Dal     ┆ M         │
│ 1   ┆ ham    ┆ D     ┆ bean    ┆ Curry   ┆ M         │
│ 1   ┆ egg    ┆ E     ┆ null    ┆ Dish    ┆ M         │
└─────┴────────┴───────┴─────────┴─────────┴───────────┘

In the case of df a infringement report should be created with the following dedicated output:

shape: (2, 8)
┌───────┬─────┬────────┬───────┬─────────┬─────────┬───────────┬─────────────────────────┐
│ index ┆ ID  ┆ column ┆ table ┆ value_a ┆ value_b ┆ mandatory ┆ warning                 │
│ ---   ┆ --- ┆ ---    ┆ ---   ┆ ---     ┆ ---     ┆ ---       ┆ ---                     │
│ i64   ┆ str ┆ str    ┆ str   ┆ str     ┆ str     ┆ str       ┆ str                     │
╞═══════╪═════╪════════╪═══════╪═════════╪═════════╪═══════════╪═════════════════════════╡
│ 0     ┆ 1   ┆ foo    ┆ A     ┆ tree    ┆ Lorem   ┆ M         ┆ Row value is not unique │
│ 1     ┆ 1   ┆ foo    ┆ A     ┆ tree    ┆ Ipsum   ┆ M         ┆ Row value is not unique │
└───────┴─────┴────────┴───────┴─────────┴─────────┴───────────┴─────────────────────────┘

The report should contain an index and a warning column. I used this line of code to identify if there are any null values in a row:

report = (df.with_row_count("index")
                .filter(pl.any(pl.col("*").is_null()) & pl.col("mandatory").eq("M"))
                .with_columns(pl.lit("Missing value detected").alias("warning"))
         )

How do I need to adapt this code so on the one hand I detect missing values and on the other hand I identify ununique rows. Maybe I create two reports and use .vstack() to combine both reports to a final one. How would you solve it?

python dataframe duplicates python-polars