What is the best way to do EDA in Databricks?

Hayley — Mon, 20 Dec 2021 22:23:29 GMT

Are there example notebooks to quickstart the exploratory data analysis?

Re: What is the best way to do EDA in Databricks?

Hayley — Mon, 20 Dec 2021 22:25:21 GMT

A quick way to start exploratory data analysis is to use the EDA notebook that is created when you use Databricks AutoML. Then you can use the notebook generated as is, or as a starting point for modeling. You’ll need a cluster with Databricks Runtime 10.0 ML and above.

To get the EDA notebook:

go to the Machine Learning Persona>select Start AutoML on the homepage
select your cluster
the type of model you are doing (Classification, Regression, Forecasting)
your training data from a database your metastore
select field that is your prediction target
click Start AutoML

When the model starts running, it will autogenerate and EDA file based on a sample of your data. Because you are using spark 3.2.x on this runtime, the koalas library is merged with Pandas to get better scalability. This notebook uses the pandas-profiling library, so you can edit the options of the library for additional analysis. In addition to profiling, you get feature interactions, correlations, and missing values. Even if you plan on using a different runtime for your modeling in production, this is a handy shortcut for EDA.

topic What is the best way to do EDA in Databricks? in Data Engineering

What is the best way to do EDA in Databricks?

Re: What is the best way to do EDA in Databricks?