A quick way to start exploratory data analysis is to use the EDA notebook that is created when you use Databricks AutoML. Then you can use the notebook generated as is, or as a starting point for modeling. You’ll need a cluster with Databricks Runtime 10.0 ML and above.
To get the EDA notebook:
- go to the Machine Learning Persona>select Start AutoML on the homepage
- select your cluster
- the type of model you are doing (Classification, Regression, Forecasting)
- your training data from a database your metastore
- select field that is your prediction target
- click Start AutoML
When the model starts running, it will autogenerate and EDA file based on a sample of your data. Because you are using spark 3.2.x on this runtime, the koalas library is merged with Pandas to get better scalability. This notebook uses the pandas-profiling library, so you can edit the options of the library for additional analysis. In addition to profiling, you get feature interactions, correlations, and missing values. Even if you plan on using a different runtime for your modeling in production, this is a handy shortcut for EDA.
