Currently I have the following problem. I have to check if there is an infringement if the column values ID, table and value_a are not unique.
import polars as pl
df = pl.DataFrame(
{
"ID": ["1", "1", "1", "1", "1"],
"column": ["foo", "foo", "bar", "ham", "egg"],
"table": ["A", "A", "C", "D", "E"],
"value_a": ["tree", tree, None, "bean", None,],
"value_b": ["Lorem", "Ipsum", "Dal", "Curry", "Dish",],
"mandatory": ["M", "M", "M", "CM", "M"]
}
)
print(df)
shape: (5, 6)
┌─────┬────────┬───────┬─────────┬─────────┬───────────┐
│ ID ┆ column ┆ table ┆ value_a ┆ value_b ┆ mandatory │
│ --- ┆ --- ┆ --- ┆ --- ┆ --- ┆ --- │
│ str ┆ str ┆ str ┆ str ┆ str ┆ str │
╞═════╪════════╪═══════╪═════════╪═════════╪═══════════╡
│ 1 ┆ foo ┆ A ┆ tree ┆ Lorem ┆ M │
│ 1 ┆ foo ┆ B ┆ tree ┆ Ipsum ┆ M │
│ 1 ┆ bar ┆ C ┆ null ┆ Dal ┆ M │
│ 1 ┆ ham ┆ D ┆ bean ┆ Curry ┆ M │
│ 1 ┆ egg ┆ E ┆ null ┆ Dish ┆ M │
└─────┴────────┴───────┴─────────┴─────────┴───────────┘
In the case of df a infringement report should be created with the following dedicated output:
shape: (2, 8)
┌───────┬─────┬────────┬───────┬─────────┬─────────┬───────────┬─────────────────────────┐
│ index ┆ ID ┆ column ┆ table ┆ value_a ┆ value_b ┆ mandatory ┆ warning │
│ --- ┆ --- ┆ --- ┆ --- ┆ --- ┆ --- ┆ --- ┆ --- │
│ i64 ┆ str ┆ str ┆ str ┆ str ┆ str ┆ str ┆ str │
╞═══════╪═════╪════════╪═══════╪═════════╪═════════╪═══════════╪═════════════════════════╡
│ 0 ┆ 1 ┆ foo ┆ A ┆ tree ┆ Lorem ┆ M ┆ Row value is not unique │
│ 1 ┆ 1 ┆ foo ┆ A ┆ tree ┆ Ipsum ┆ M ┆ Row value is not unique │
└───────┴─────┴────────┴───────┴─────────┴─────────┴───────────┴─────────────────────────┘
The report should contain an index and a warning column. I used this line of code to identify if there are any null values in a row:
report = (df.with_row_count("index")
.filter(pl.any(pl.col("*").is_null()) & pl.col("mandatory").eq("M"))
.with_columns(pl.lit("Missing value detected").alias("warning"))
)
How do I need to adapt this code so on the one hand I detect missing values and on the other hand I identify ununique rows. Maybe I create two reports and use .vstack() to combine both reports to a final one. How would you solve it?