cancel
Showing results forย 
Search instead forย 
Did you mean:ย 
Data Engineering
Join discussions on data engineering best practices, architectures, and optimization strategies within the Databricks Community. Exchange insights and solutions with fellow data engineers.
cancel
Showing results forย 
Search instead forย 
Did you mean:ย 

What is the best way to do EDA in Databricks?

Hayley
New Contributor III

Are there example notebooks to quickstart the exploratory data analysis?

2 REPLIES 2

Hayley
New Contributor III

A quick way to start exploratory data analysis is to use the EDA notebook that is created when you use Databricks AutoML. Then you can use the notebook generated as is, or as a starting point for modeling. Youโ€™ll need a cluster with Databricks Runtime 10.0 ML and above.

To get the EDA notebook:

  • go to the Machine Learning Persona>select Start AutoML on the homepage
  • select your cluster
  • the type of model you are doing (Classification, Regression, Forecasting)
  • your training data from a database your metastore
  • select field that is your prediction target
  • click Start AutoML

When the model starts running, it will autogenerate and EDA file based on a sample of your data. Because you are using spark 3.2.x on this runtime, the koalas library is merged with Pandas to get better scalability. This notebook uses the pandas-profiling library, so you can edit the options of the library for additional analysis. In addition to profiling, you get feature interactions, correlations, and missing values. Even if you plan on using a different runtime for your modeling in production, this is a handy shortcut for EDA.

Hi @Hayley Hornโ€‹ , Thank you for the amazing informative post.

Connect with Databricks Users in Your Area

Join a Regional User Group to connect with local Databricks users. Events will be happening in your city, and you wonโ€™t want to miss the chance to attend and share knowledge.

If there isnโ€™t a group near you, start one and help create a community that brings people together.

Request a New Group