Hi @manojvas, Databricks has a built-in data profiling feature that you can use.
When you use the display(<dataframe>)
command in Scala or Python or run a SQL query, the results pane shows a new tab, "Data Profiles," that presents an interactive tabular and graphic summary of the DataFrame or table.
You can also use the Databricks utilities command.
dbutils.data.summarize
If you are experiencing errors, it might be due to various reasons, such as standard errors in notebooks, issues with Databricks Connect, or problems related to your Spark or Python setup.