Get started with an exploratory analysis on Databricks using Sample Data

[DatabricksGuide]

Getting started with Databricks - Exploratory analysis

This guide walks you through using a Databricks notebook to query sample data stored in Unity Catalog using Python and then visualize the query results in the notebook.

This is a beginner’s tutorial with hands-on instructions to execute in your own Databricks workspace. You can request a free 14-day trial. 

Step 1: Access sample data

1. Click Catalog in the sidebar to open the Catalog Explorer. This is the primary UI for exploring and managing data, including schemas, tables, models, and other data objects.
2. Click the samples catalog. This Databricks-managed catalog contains two sample datasets for your exploration: nyctaxi and tpch. 
3. Click the nyctaxi schema, and then open the trips table.

Step 2: Create a notebook

1. Use the Create button in the top-right to create a notebook with a pre-populated query of the trips table. New, blank queries and notebooks can also be created using the +New button from the top left of the screen. 
2. If you created a blank notebook or query, you can query the table you just created using the SQL SELECT statement SELECT * from <catalog-name>.<schema-name>.<table-name>. If you choose to run your analysis in a notebook, be sure to change the notebook language to SQL by changing the language selection toggle in the top pane of the notebook. 
3. Here are some sample commands to try out in Python:

import pandas as pd

# Read the sample NYC Taxi Trips dataset and load it into a PySpark DataFrame.Convert  PySpark DataFrame to Pandas DataFrame
df = spark.read.table('samples.nyctaxi.trips')
pdf = df.toPandas()

# Select the 10 most expensive and the 10 longest trips based on fare_amount
most_expensive_trips = pdf.nlargest(10,'fare_amount')
longest_trips = pdf.nlargest(10,'trip_distance')

Step 3: Use the Databricks Assistant for code suggestions

1. Notebooks come equipped with the context-aware Databricks Assistant, which can help generate, explain, and fix code using natural language.
2. To use the assistant, create a new cell and click CMD+I or click the Assistant icon on the top right corner of the new cell.
3. Enter a prompt for the Assistant to provide code suggestions. Here are some sample prompts:
1. What is the most common pickup zip across all trips?
2. What is the minimum, maximum, and average fare amount across all trips?
   
   
4. Press Return or the submit button to submit the prompt and watch the Assistant suggest code to answer the prompt. Click “Accept” to save the code suggestion and run the cell to view the results!

Step 4: Visualize the data

1. You can visualize the data in your table from the results of the query by clicking the + button at the top of the results experience and completing the visualization builder dialog.
2. Select your preferred visualization type, and fill out the chart values to prepare the chart.

Step 5: Share its results with non-Databricks users

Add users to your Databricks Workspace

1. In the top bar of the Databricks workspace, click your username and then click Settings.
2. In the sidebar, click Identity and Access.
3. Next to Users, click Manage.
4. Click Add user, and then click Add new.
5. Enter the user’s email address, and then click Add.

Continue to add as many users to your account as you would like. New users receive an email prompting them to set up their account.

Share the notebook with colleagues

1. To manage access to the notebook, click Share at the top of the notebook to open the permissions dialog. 
2. Share your notebook with any colleague by adding the “All Users” group to the notebook’s access list with “Can View” or “Can Run” permission and sending your colleague the notebook’s URL, which can be copied to your clipboard using the “Copy link” button.

Next steps

1. To learn about adding data from CSV files to Unity Catalog and visualize data, see Get started: Import and visualize CSV data from a notebook.
2. To learn how to load data into Databricks using Apache Spark, see Tutorial: Load and transform data using Apache Spark DataFrames.
3. To learn more about ingesting data into Databricks, see Ingest data into a Databricks lakehouse.
4. To learn more about querying data with Databricks, see Query data.
5. To learn more about visualizations, see Visualizations in Databricks notebooks.