Tracking changes in data distribution by using pyspark

sridhar0109
New Contributor

Hi All,

I'm working on creating a data quality dashboard. I've created few rules like checking nulls in a column, checking for data type of the column , removing duplicates etc.

We follow medallion architecture and are applying these data quality checks on bronze table and insert rows which pass the data quality checks as mentioned above.

Now, I want to track distribution of a column over a period of time like for example: I have sales data for different car models ,then a distribution of the sales of each car model over a period of time.

Could you please suggest if there are any out of box libraries available to achieve this task?

Thanks!