Data-quality help: Save Data Profile dbutils.data.summarize(df) to table

[Kash]

Hi there,

We would like to create a data quality database that helps us understand how complete our data is. We would like to run a job each day that basically outputs the same table data as dbutils.data.summarize(df) for a given table and save it to databricks.

Any ideas on how we could do that?

Thanks,

Avkash

[daniel_sahal]

From what I know there's no easy way to save dbutils.data.summarize() into a df.

You can still create your custom python/pyspark code to profile your data and save the output.