Ydata-profiling on Spark cluster -working for you?

[manojvas]

I am trying to profile my dataset using ydata-profiling.

I constantly run into errors, even with simple datasets on my spark cluster.

Profiling in Spark cluster erroring out · Issue #1350 · ydataai/ydata-profiling (github.com)

Has anyone got ydata-profiling to work on their databricks cluster?

[Kaniz_Fatma · Accepted solution]

Hi @manojvas, Databricks has a built-in data profiling feature that you can use.

When you use the display(<dataframe>) command in Scala or Python or run a SQL query, the results pane shows a new tab, "Data Profiles," that presents an interactive tabular and graphic summary of the DataFrame or table.

You can also use the Databricks utilities command.

 dbutils.data.summarize 

If you are experiencing errors, it might be due to various reasons, such as standard errors in notebooks, issues with Databricks Connect, or problems related to your Spark or Python setup.

[manojvas]

Thank you @Kaniz_Fatma