https://community.databricks.com/t5/data-engineering/data-quality-help-save-data-profile-dbutils-data-summarize-df-to/m-p/15597#M9905 <P>Hi there,</P><P></P><P>We would like to create a data quality database that helps us understand how complete our data is. We would like to run a job each day that basically outputs the same table data as dbutils.data.summarize(df) for a given table and save it to databricks.</P><P></P><P>Any ideas on how we could do that?</P><P></P><P>Thanks,</P><P></P><P>Avkash</P> Tue, 20 Dec 2022 20:05:31 GMT Kash 2022-12-20T20:05:31Z https://community.databricks.com/t5/data-engineering/data-quality-help-save-data-profile-dbutils-data-summarize-df-to/m-p/15597#M9905 <P>Hi there,</P><P></P><P>We would like to create a data quality database that helps us understand how complete our data is. We would like to run a job each day that basically outputs the same table data as dbutils.data.summarize(df) for a given table and save it to databricks.</P><P></P><P>Any ideas on how we could do that?</P><P></P><P>Thanks,</P><P></P><P>Avkash</P> Tue, 20 Dec 2022 20:05:31 GMT https://community.databricks.com/t5/data-engineering/data-quality-help-save-data-profile-dbutils-data-summarize-df-to/m-p/15597#M9905 Kash 2022-12-20T20:05:31Z https://community.databricks.com/t5/data-engineering/data-quality-help-save-data-profile-dbutils-data-summarize-df-to/m-p/15598#M9906 <P>From what I know there's no easy way to save dbutils.data.summarize() into a df.</P><P>You can still create your custom python/pyspark code to profile your data and save the output. </P> Thu, 22 Dec 2022 13:17:20 GMT https://community.databricks.com/t5/data-engineering/data-quality-help-save-data-profile-dbutils-data-summarize-df-to/m-p/15598#M9906 daniel_sahal 2022-12-22T13:17:20Z