New knowledge base articles in Learning & Certification

Getting started with Databricks - Deploy an LLM chatbot using Databricks Vector Search and Foundation Model API

DatabricksGuide — Thu, 15 Aug 2024 17:53:44 GMT

Getting started with Databricks - Deploy an LLM chatbot using Databricks Vector Search and Foundation Model API

This guide demonstrates how to build a simple Retrieval-Augmented Generation (RAG) application using Databricks Vector Search, a vector database seamlessly integrated with Databricks' Foundation Model API (FMAPI) embedding models.

RAG is a widely adopted architecture for creating natural-language interfaces that allow users to interact with organizational data effectively. This tutorial will walk you through setting up a vector index, loading text data, querying the database, building a prompt for a large language model (LLM), and finally querying the LLM using the FMAPI. All these steps can be completed in just 30 minutes.

This is a beginner’s guide with hands-on instructions that you can follow in your own Databricks workspace.

To learn more about how Databricks Vector Search works, refer to the documentation here.
For more information on querying models via the Foundation Model APIs, see the documentation here.

Setup

First, we will install the necessary libraries and set up a temporary catalog/schema/table for this example.

In [ ]: %pip install --upgrade databricks-vectorsearch databricks-genai-inference dbutils.library.restartPython()

Define catalog, table, endpoint, and index names

In [ ]:
CATALOG = "workspace" DB='vs_demo' SOURCE_TABLE_NAME = "documents" SOURCE_TABLE_FULLNAME=f"{CATALOG}.{DB}.{SOURCE_TABLE_NAME}"

Create Catalog, Schema, and Table

A Databricks Vector Search Index is created from a Delta Table. The source Delta Table includes the data we ultimately want to index and search with the vector database. In this cell, we create the catalog, schema, and source table from which we will create the vector database.

In [ ]: # Set up schema/volume/table from pyspark.sql.types import StructType, StructField, StringType, ArrayType, FloatType spark.sql(f"CREATE SCHEMA IF NOT EXISTS {CATALOG}.{DB}") spark.sql( f"""CREATE TABLE IF NOT EXISTS {SOURCE_TABLE_FULLNAME} ( id STRING, text STRING, date DATE, title STRING ) USING delta TBLPROPERTIES ('delta.enableChangeDataFeed' = 'true') """ )

Set up the Vector Database

Next, we set up the vector database. There are three key steps:

Initialize the vector search client
Create the endpoint
Create the index using the source Delta table we created earlier and the bge-large-en embeddings model from the Foundation Model API

Initialize the Vector Search Client

In [ ]:
from databricks.vector_search.client import VectorSearchClient vsc = VectorSearchClient()

Create the Endpoint

The cell below will check if the endpoint already exists and create it if it does not.

In [ ]: VS_ENDPOINT_NAME = 'vs_endpoint' if vsc.list_endpoints().get('endpoints') == None or not VS_ENDPOINT_NAME in [endpoint.get('name') for endpoint in vsc.list_endpoints().get('endpoints')]: print(f"Creating new Vector Search endpoint named {VS_ENDPOINT_NAME}") vsc.create_endpoint(VS_ENDPOINT_NAME) else: print(f"Endpoint {VS_ENDPOINT_NAME} already exists.") vsc.wait_for_endpoint(VS_ENDPOINT_NAME, 600)

Create the Vector Index

Now we can create the index over the Delta table we created earlier.

In [ ]: VS_INDEX_NAME = 'fm_api_examples_vs_index' VS_INDEX_FULLNAME = f"{CATALOG}.{DB}.{VS_INDEX_NAME}" if not VS_INDEX_FULLNAME in [index.get("name") for index in vsc.list_indexes(VS_ENDPOINT_NAME).get('vector_indexes', [])]: try: # set up an index with managed embeddings print("Creating Vector Index...") i = vsc.create_delta_sync_index_and_wait( endpoint_name=VS_ENDPOINT_NAME, index_name=VS_INDEX_FULLNAME, source_table_name=SOURCE_TABLE_FULLNAME, pipeline_type="TRIGGERED", primary_key="id", embedding_source_column="text", embedding_model_endpoint_name="databricks-bge-large-en" ) except Exception as e: if "INTERNAL_ERROR" in str(e): # Check if the index exists after the error occurred if VS_INDEX_FULLNAME in [index.get("name") for index in vsc.list_indexes(VS_ENDPOINT_NAME).get('vector_indexes', [])]: print(f"Index {VS_INDEX_FULLNAME} has been created.") else: raise e else: raise e else: print(f"Index {VS_INDEX_FULLNAME} already exists.")

There are a few key points to note about the specific configuration we used in this case:

We used pipeline_type="TRIGGERED". This requires us to use the index's sync() method to manually sync the source Delta table with the index. We could, alternatively, use pipeline_type="CONTINUOUS" which will automatically keep the index in sync with the source table with only seconds of latency. This approach is more costly, though, as a compute cluster must be provisioned for the continuous sync streaming pipeline.
We specified embedding_model_endpoint_name="databricks-bge-large-en". We can use any embedding model available via model serving; this is the name of the pay-per-token Foundation Model API version of databricks-bge-large-en. By passing an embedding_source_column and embedding_model_endpoint_name, we configure the index such that it will automatically use the model to generate embeddings for the texts in the text column of the source table. We do not need to manually generate embeddings.

If, however, we did want to manage embeddings manually, we could include the following arguments instead:
```
  embedding_vector_column="<embedding_column>",
  embedding_dimension=<embedding_dimension>
```
In the latter approach, we include a column for embeddings in the source delta table and embeddings are not computed automatically from the text column.

Set up some example texts

Now we set up some example texts to index.

# Some example texts from datetime import datetime smarter_overview = {"text":""" S.M.A.R.T.E.R. Initiative: Strategic Management for Achieving Results through Efficiency and Resources Introduction The S.M.A.R.T.E.R. Initiative, standing for "Strategic Management for Achieving Results through Efficiency and Resources," is a groundbreaking project aimed at revolutionizing the way our organization operates. In today's rapidly changing business landscape, achieving success demands a strategic approach that leverages resources effectively while optimizing efficiency. The S.M.A.R.T.E.R. Initiative is designed to do just that. Background As markets evolve and competition intensifies, organizations must adapt to stay relevant and profitable. Traditional methods of operation often become inefficient and costly. The S.M.A.R.T.E.R. Initiative was conceived as a response to this challenge, with the primary goal of enhancing strategic management practices to achieve better results. Objectives 1. Resource Optimization One of the key objectives of the S.M.A.R.T.E.R. Initiative is to optimize resource allocation. This involves identifying underutilized resources, streamlining processes, and reallocating resources to areas that contribute most to our strategic goals. 2. Efficiency Improvement Efficiency is at the core of the S.M.A.R.T.E.R. Initiative. By identifying bottlenecks and improving processes, we aim to reduce operational costs, shorten project timelines, and enhance overall productivity. 3. Strategic Alignment For any organization to succeed, its activities must be aligned with its strategic objectives. The S.M.A.R.T.E.R. Initiative will ensure that every action and resource allocation is in sync with our long-term strategic goals. 4. Results-driven Approach The ultimate measure of success is results. The S.M.A.R.T.E.R. Initiative will foster a results-driven culture within our organization, where decisions and actions are guided by their impact on our bottom line and strategic objectives. Key Components The S.M.A.R.T.E.R. Initiative comprises several key components: 1. Data Analytics and Insights Data is the foundation of informed decision-making. We will invest in advanced data analytics tools to gain insights into our operations, customer behavior, and market trends. These insights will guide our resource allocation and strategy. 2. Process Automation Automation will play a vital role in enhancing efficiency. Routine and repetitive tasks will be automated, freeing up our workforce to focus on more strategic activities. 3. Performance Metrics and KPIs To ensure that our efforts are aligned with our objectives, we will establish a comprehensive set of Key Performance Indicators (KPIs). Regular monitoring and reporting will provide visibility into our progress. 4. Training and Development Enhancing our workforce's skills is essential. We will invest in training and development programs to equip our employees with the knowledge and tools needed to excel in their roles. Implementation Timeline The S.M.A.R.T.E.R. Initiative will be implemented in phases over the next three years. This phased approach allows for a smooth transition and ensures that each component is integrated effectively into our operations. Conclusion The S.M.A.R.T.E.R. Initiative represents a significant step forward for our organization. By strategically managing our resources and optimizing efficiency, we are positioning ourselves for sustained success in a competitive marketplace. This initiative is a testament to our commitment to excellence and our dedication to achieving exceptional results. As we embark on this journey, we look forward to the transformative impact that the S.M.A.R.T.E.R. Initiative will have on our organization and the benefits it will bring to our employees, customers, and stakeholders. """, "title": "Project Kickoff", "date": datetime.strptime("2024-01-16", "%Y-%m-%d")} smarter_kpis = {"text": """S.M.A.R.T.E.R. Initiative: Key Performance Indicators (KPIs) Introduction The S.M.A.R.T.E.R. Initiative (Strategic Management for Achieving Results through Efficiency and Resources) is designed to drive excellence within our organization. To measure the success and effectiveness of this initiative, we have established three concrete and measurable Key Performance Indicators (KPIs). This document outlines these KPIs and their associated targets. Key Performance Indicators (KPIs) 1. Resource Utilization Efficiency (RUE) Objective: To optimize resource utilization for cost-efficiency. KPI Definition: RUE will be calculated as (Actual Resource Utilization / Planned Resource Utilization) * 100%. Target: Achieve a 15% increase in RUE within the first year. 2. Time-to-Decision Reduction (TDR) Objective: To streamline operational processes and reduce decision-making time. KPI Definition: TDR will be calculated as (Pre-Initiative Decision Time - Post-Initiative Decision Time) / Pre-Initiative Decision Time. Target: Achieve a 20% reduction in TDR for critical business decisions. 3. Strategic Goals Achievement (SGA) Objective: To ensure that organizational activities align with strategic goals. KPI Definition: SGA will measure the percentage of predefined strategic objectives achieved. Target: Achieve an 80% Strategic Goals Achievement rate within two years. Conclusion These three KPIs, Resource Utilization Efficiency (RUE), Time-to-Decision Reduction (TDR), and Strategic Goals Achievement (SGA), will serve as crucial metrics for evaluating the success of the S.M.A.R.T.E.R. Initiative. By tracking these KPIs and working towards their targets, we aim to drive efficiency, optimize resource utilization, and align our actions with our strategic objectives. This focus on measurable outcomes will guide our efforts towards achieving excellence within our organization.""", "title": "Project KPIs", "date": datetime.strptime("2024-01-16", "%Y-%m-%d")}

Chunk the texts

Typically, when using a vector database for retrieval-augmented generation (RAG) tasks, we break the texts apart into smaller (and sometimes overlapping) chunks to return focused and relevant information without returning an excessive amount of text.

In the code below, we break the sample texts above into shorter overlapping text chunks.

import re

def chunk_text(text, chunk_size, overlap):
    words = text.split()
    chunks = []
    index = 0

    while index < len(words):
        end = index + chunk_size
        while end < len(words) and not re.match(r'.*[.!?]\s*$', words[end]):
            end += 1
        chunk = ' '.join(words[index:end+1])
        chunks.append(chunk)
        index += chunk_size - overlap

    return chunks

chunks = []
documents = [smarter_overview, smarter_kpis]

for document in documents:
    for i, c in enumerate(chunk_text(document["text"], 150, 25)):
        chunk = {}
        chunk["text"] = c
        chunk["title"] = document["title"]
        chunk["date"] = document["date"]
        chunk["id"] = document["title"] + "_" + str(i)

        chunks.append(chunk)

Insert the text chunks into the source delta table

Now we save the chunks, along with some metadata (a document title, date, and a unique id) to the source delta table.

from pyspark.sql.types import StructType, StructField, StringType, ArrayType, FloatType, DateType

schema = StructType(
    [
        StructField("id", StringType(), True),
        StructField("text", StringType(), True),
        StructField("title", StringType(), True),
        StructField("date", DateType(), True),
    ]
)

if chunks:
    result_df = spark.createDataFrame(chunks, schema=schema)
    result_df.write.format("delta").mode("append").saveAsTable(
        SOURCE_TABLE_FULLNAME
    )
     

Sync the Vector Search Index

Because we specified pipeline_type="TRIGGERED" when configuring the Delta Index, we still need to manually tell the index to sync with the delta table. This will take a few minutes.

This will not work if the index is not ready yet. We use the wait_until_ready method to wait until the index is ready.

In [ ]:
# Sync 

index = vsc.get_index(endpoint_name=VS_ENDPOINT_NAME, index_name=VS_INDEX_FULLNAME) index.sync()

Query the Vector Index

Now that we have added our text chunks to the source delta table and synced it with the Vector Search index, we're ready to query the index! We do this with the index.similarity_search() method.

The columns argument takes a list of the columns we want returned; in this case, we request the text and title columns.

NOTE: If the cell below does not return any results, wait a couple of minutes and try again. The index may still be syncing.

In [ ]: # query index.similarity_search(columns=["text", "title"], query_text="What is the TDR Target for the SMARTER initiative?", num_results = 3)

Answering Questions about the Output

from databricks_genai_inference import ChatSession

chat = ChatSession(model="databricks-meta-llama-3-70b-instruct",
                   system_message="You are a helpful assistant.",
                   max_tokens=128)

First, let's ask a question without RAG.

chat.reply("What is the TDR Target for the SMARTER initiative?")
chat.last

As you can see, its response has nothing to do with the documents we processed above.

Now let's see what kind of reply we get when we provide context from vector search.

# reset history chat = ChatSession(model="databricks-meta-llama-3-70b-instruct", system_message="You are a helpful assistant. Answer the user's question based on the provided context.", max_tokens=128) # get context from vector search raw_context = index.similarity_search(columns=["text", "title"], query_text="What is the TDR Target for the SMARTER initiative?", num_results = 3) context_string = "Context:\n\n" for (i,doc) in enumerate(raw_context.get('result').get('data_array')): context_string += f"Retrieved context {i+1}:\n" context_string += doc[0] context_string += "\n\n" chat.reply(f"User question: What is the TDR Target for the SMARTER initiative?\n\nContext: {context_string}") chat.last

It is now able to answer based on the provided context.

Congratulations! Your chatbot is ready.

Additional information

Using the UI

Most of the Vector Database management steps above can be done via the UI: you can create an endpoint, create an index, sync the index, and more via the UI in the Databricks Catalog Explorer.

Experimenting in the AI Playground

The Databricks AI Playground provides a GUI for quickly experimenting with LLMs available via the FMAPI, enabling you to compare the outputs of those models and determine which model best serves your needs.

Get started with an exploratory analysis on Databricks using Sample Data

DatabricksGuide — Mon, 26 Aug 2024 16:53:19 GMT

Getting started with Databricks - Exploratory analysis

This guide walks you through using a Databricks notebook to query sample data stored in Unity Catalog using Python and then visualize the query results in the notebook.

This is a beginner’s tutorial with hands-on instructions to execute in your own Databricks workspace. You can request a free 14-day trial.

Step 1: Access sample data

Click Catalog in the sidebar to open the Catalog Explorer. This is the primary UI for exploring and managing data, including schemas, tables, models, and other data objects.
Click the samples catalog. This Databricks-managed catalog contains two sample datasets for your exploration: nyctaxi and tpch.
Click the nyctaxi schema, and then open the trips table.

Step 2: Create a notebook

Use the Create button in the top-right to create a notebook with a pre-populated query of the trips table. New, blank queries and notebooks can also be created using the +New button from the top left of the screen.
If you created a blank notebook or query, you can query the table you just created using the SQL SELECT statement SELECT * from <catalog-name>.<schema-name>.<table-name>. If you choose to run your analysis in a notebook, be sure to change the notebook language to SQL by changing the language selection toggle in the top pane of the notebook.
Here are some sample commands to try out in Python:

import pandas as pd # Read the sample NYC Taxi Trips dataset and load it into a PySpark DataFrame.Convert PySpark DataFrame to Pandas DataFrame df = spark.read.table('samples.nyctaxi.trips') pdf = df.toPandas() # Select the 10 most expensive and the 10 longest trips based on fare_amount most_expensive_trips = pdf.nlargest(10,'fare_amount') longest_trips = pdf.nlargest(10,'trip_distance')

Step 3: Use the Databricks Assistant for code suggestions

Notebooks come equipped with the context-aware Databricks Assistant, which can help generate, explain, and fix code using natural language.
To use the assistant, create a new cell and click CMD+I or click the Assistant icon on the top right corner of the new cell.
Enter a prompt for the Assistant to provide code suggestions. Here are some sample prompts:

What is the most common pickup zip across all trips?
What is the minimum, maximum, and average fare amount across all trips?

Press Return or the submit button to submit the prompt and watch the Assistant suggest code to answer the prompt. Click “Accept” to save the code suggestion and run the cell to view the results!

Step 4: Visualize the data

You can visualize the data in your table from the results of the query by clicking the + button at the top of the results experience and completing the visualization builder dialog.
Select your preferred visualization type, and fill out the chart values to prepare the chart.

Step 5: Share its results with non-Databricks users

Add users to your Databricks Workspace

In the top bar of the Databricks workspace, click your username and then click Settings.
In the sidebar, click Identity and Access.
Next to Users, click Manage.
Click Add user, and then click Add new.
Enter the user’s email address, and then click Add.

Continue to add as many users to your account as you would like. New users receive an email prompting them to set up their account.

Share the notebook with colleagues

To manage access to the notebook, click Share at the top of the notebook to open the permissions dialog.
Share your notebook with any colleague by adding the “All Users” group to the notebook’s access list with “Can View” or “Can Run” permission and sending your colleague the notebook’s URL, which can be copied to your clipboard using the “Copy link” button.

Next steps

To learn about adding data from CSV files to Unity Catalog and visualize data, see Get started: Import and visualize CSV data from a notebook.
To learn how to load data into Databricks using Apache Spark, see Tutorial: Load and transform data using Apache Spark DataFrames.
To learn more about ingesting data into Databricks, see Ingest data into a Databricks lakehouse.
To learn more about querying data with Databricks, see Query data.
To learn more about visualizations, see Visualizations in Databricks notebooks.

Getting Started with Databricks - Building a Forecasting Model on Databricks

DatabricksGuide — Mon, 12 Aug 2024 18:15:10 GMT

Building a Forecasting Model on Databricks: A Step-by-Step Guide

This guide offers a detailed, step-by-step approach for building a forecasting model on Databricks. By leveraging the power of Databricks, you will unlock new potentials in your data workflows, streamline model deployment processes, and optimize your forecasting accuracy with state-of-the-art tools and practices.

*It assumes you have a basic familiarity with Databricks notebooks, PySpark, and common machine learning concepts.

What Will You Learn?

This guide is structured to provide a step-by-step approach to advanced forecasting and model deployment, incorporating best practices and leveraging modern tools such as MLflow. By the end of this guide, you will be equipped with the knowledge to:

Effectively Prepare and Process Data: Understand how to select, clean, and prepare your data for forecasting, emphasizing the importance of feature engineering and data quality.
Build Robust Forecasting Models: Dive into the selection of sophisticated forecasting models that cater to various data characteristics.
Master Model Logging with MLflow: Learn the intricacies of model logging, enabling efficient tracking of model versions, parameters, and performance metrics.
Deploy Models with Confidence: Navigate through the model serving landscape, understanding different deployment strategies for making your models accessible for predictions.
Visualize and Interpret Model Predictions: Discover techniques for visualizing forecasting results, allowing you to compare predicted values against actual historical data and assess model performance visually.
Best Practices for Forecasting and Model Deployment: Throughout the guide, insights into best practices will prepare you to tackle common challenges and make informed decisions.

Data Selection

Data Selection is a crucial initial step for successful time series forecasting. Here's a breakdown of the key aspects to consider

Identify the Forecasting Goal: What are you trying to predict? Sales, website traffic, resource utilization? Clearly define the target variable you want to forecast.
Data Frequency: Choose a data frequency that aligns with your forecasting needs. Daily, weekly, monthly, or even hourly data might be suitable depending on the problem.
Data Sufficiency: Make sure you have enough data to train and validate your forecasting model. The amount of data required depends on the model complexity (e.g., LSTMs typically require more data than ARIMA).

A successful forecasting model relies on a well-structured dataset. For this tutorial, your data should ideally at least have the following columns:

Date: Timestamp for each observation.
Metric: The specific aspect you’re forecasting (e.g., sales, website traffic).
Metric Value: The numerical data point you aim to predict (e.g., number of users, sales amount).

Ensure the data is at the correct frequency (daily, weekly, etc.) and that you have enough data to capture seasonal patterns or trends relevant to your goals.

Ingest Setup Steps [20 minutes]

Step 0: Check your required prerequisites

This section of the guide assumes you have the following:

You have a Databricks workspace up and running
Your account has Unity Catalog enabled. UC is enabled by default
You have admin permissions for your Databricks account
You have access to your AWS account and data residing in a S3 storage bucket

Step 1: Access and start your warehouse

To get started with Databricks, you need to kickstart your starter warehouse. This compute resource will let you query and explore data on Databricks. You can also use Serverless Warehouse to use this guide.

Note: Only users with workspace administrator permissions are able to start the starter warehouse. If you don’t have admin permissions, please contact your administrator to get access to the starter warehouse or to create a new serverless warehouse.

Step 2: Connect your workspace to data sources

To connect your Databricks workspace to your cloud storage, you need to create an external location. An external location is an object that combines a cloud storage path with the credential that authorizes access to the storage path.

Watch and follow along in your workspace as you go!

In your Databricks workspace, click Catalog on the sidebar.
At the top of the page, click + Add.
Click Add an external location.
Databricks recommends using the AWS Quickstart, which ensures that your workspace is given the correct permissions on the bucket.
In Bucket Name, enter the name of the bucket you want to import data from.
Click Generate New Token and copy the token.
Click Launch in Quickstart.
In your AWS console, enter the copied token in the Databricks Personal Access Token field.
Select the I acknowledge that AWS CloudFormation might create IAM resources with custom names checkbox.
Click Create stack. To see the external locations in your workspace, click Catalog in the sidebar, at the bottom of the left navigation pane click External Data, and then click External Locations. Your new external location will have a name using the following syntax: db_s3_external_databricks-S3-ingest-<id>.
Test your connection

To verify external locations have functioning connections, do the following:

Click the external location you want to test.
Click Test connection.

For help, contact onboarding-help@databricks.com or your Databricks account team directly.

Step 3: Add your data to Databricks

Now that your workspace has a connection to your S3 bucket, you can add your data.

Part of this step is choosing where to put your data. Databricks has a three-level namespace that organizes your data (catalog.schema.table). For this exercise, you import the data into the default catalog named after your workspace.

In the sidebar of your Databricks workspace, click New > Add data.
Click Amazon S3.
Select your external location from the drop-down menu.
Select all the files you want to add to your Databricks catalog.
Click Preview table.
Select the default catalog (named after your workspace), the default schema, and then enter a name for your table.
Click Create Table.

You can now use Catalog Explorer in your workspace to see your data in Databricks.

Forecasting on Databricks

Once you have selected your Data, there are 2 primary ways to build a Forecasting Model on Databricks.

Databricks AutoML: This is a user-friendly option, especially for those new to forecasting or machine learning. Databricks AutoML automates the process of selecting and training the most suitable forecasting model for your data.
Databricks Notebook: This is designed for users with experience in machine learning and time series forecasting. It offers more control and flexibility for building customized models.

A) Forecasting Model Using AutoML

Databricks AutoML is a valuable tool for getting started with time series forecasting on Databricks. It streamlines the process, reduces development time, and provides a solid baseline model. Use the following steps to set up an AutoML Model.

Set up the Forecasting Problem

You can set up a forecasting problem using the AutoML UI with the following steps:

Click on New -> AutoML Experiment
In the Compute field, select a cluster running Databricks Runtime 10.0 ML or above.
From the ML problem type drop-down menu, select Forecasting.
Under Dataset, click Browse. Navigate to the table you want to use and click Select. The table schema appears.
Click in the Prediction target field. A dropdown menu appears listing the columns shown in the schema. Select the column you want the model to predict.
Click in the Time column field. A drop-down appears showing the dataset columns that are of type timestamp or date. Select the column containing the time periods for the time series.
For multi-series forecasting, select the column(s) that identify the individual time series from the Time series identifiers drop-down. AutoML groups the data by these columns as different time series and trains a model for each series independently.
In the Forecast horizon and frequency fields, specify the number of time periods into the future for which AutoML should calculate forecasted values.
In Databricks Runtime 10.5 ML and above, you can save prediction results. To do so, specify a database in the Output Database field. Click Browse and select a database from the dialog. AutoML writes the prediction results to a table in this database.
The Experiment name field shows the default name. To change it, type the new name in the field.

Register the Best Model

Once your AutoML experiment finishes, identify the best performing model run based on your chosen metric.
In the Experiments UI, navigate to the run details of the best model.
Look for a "Register Model" button or option.
Select to register via ’Unity Catalog’.
Run the provided code in a notebook to register the model.

Create a Serving Endpoint

Click on New -> Serving Endpoint
Initiate the creation of the endpoint. In the Name field, provide a descriptive name for your endpoint.
Under the Served entities section, click on the Entity field.
Choose Unity Catalog as the source and select your model by choosing the appropriate catalog, schema and model name.
Select the specific model and its version you want to deploy.
Select the appropriate Compute Type (CPU for this model) based on your model's requirements.
Select ‘Small’ as the Compute Scale-out.
Once you've configured the settings, click the Create button.

Monitor Endpoint Status: The Serving endpoints page will display your newly created endpoint. The initial state will likely be "Pending" as deployment progresses. You can refresh the page to monitor the endpoint's status until it becomes "Active."

B) Forecasting Model Using Notebook

Select Data

#Install Dependencies !pip install prophet !pip install databricks-sdk==0.12.0 !pip install mlflow !pip install grpcio !pip install grpcio-status !pip install -U "pandas<2.0.0" dbutils.library.restartPython() #Select Data query = f"SELECT <date>, <metric>, <metric_value> FROM <catalog_name>.<schema_name>.<table_name> where is_customer = 'true' and <date> > '2020-01-01' order by date desc" #Update the query to get data from your table df = spark.sql(query) # Choose a single metric to make the calculations simpler df = df.filter(df.<metric> == "xxxxxxxxxxxxxxxxxxxxx") #Replace <metric> with your metric name df.show(5)

Data Preparation

Effectively preparing your data is a foundational step in the forecasting process. Proper preparation ensures the accuracy and reliability of your model's predictions.

Missing Values: Identify and address missing values in your data. Common strategies include deletion (if minimal), imputation (filling in missing values with statistical methods or previous observations), or interpolation (estimating missing values based on surrounding data points).
Outliers: Identify and handle outliers, which are extreme data points that can significantly distort your forecasts. You can choose to remove outliers if they are truly erroneous or winsorize them (capping their values to a certain threshold).
Time Consistency: Ensure your data has consistent timestamps and that the time steps are evenly spaced (e.g., daily data points should be recorded at the same time each day).
Feature Engineering: Create new features from existing ones if it can improve the forecasting model's performance. This might involve calculating rolling averages, seasonality indicators, or lag features (past values of the target variable).

from pyspark.sql.functions import col, lit # Dropping rows with missing values in the 'metric_value' column cleaned_df = df.na.drop(subset=["<metric_value>"]) #Replace <metric_value> with your metric value column name cleaned_df.show(5) # Calculating IQR and defining bounds for outliers quartiles = cleaned_df.approxQuantile("<metric_value>", [0.25, 0.75], 0.05) #Replace <metric_value> with your metric value column name IQR = quartiles[1] - quartiles[0] lower_bound = 0 upper_bound = quartiles[1] + 1.5 * IQR # Filtering out outliers no_outliers_df = cleaned_df.filter( (col("<metric_value>") > lit(lower_bound)) #Replace <metric_value> with your metric value column name & (col("<metric_value>") <= lit(upper_bound)) #Replace <metric_value> with your metric value column name ) # Showing the updated DataFrame no_outliers_df.show(5)

Model Selection

The model you choose will depend on the nature of your data and the specific forecasting problem you’re trying to solve. Take into consideration different data characteristics such as Frequency (daily vs weekly), granularity (hourly vs daily sales), seasonality, and any other external factors such as holidays, promotions etc. Machine learning methods:

Prophet: User-friendly and specifically designed for time series forecasting, offering built-in seasonality and holiday handling.
ARIMA: A classical statistical method for time series forecasting, capturing short-term patterns and trends in stationary data through autocorrelation, differencing, and moving average components.
LSTMs (Long Short-Term Memory): Powerful for capturing complex relationships and long-term dependencies in time series data.

For the scope of this tutorial, Prophet will serve as our primary model. Prophet stands out for its user-friendly nature and robust handling of various time series forecasting challenges, making it a versatile option for a wide range of applications.

Model Choice Justification

When deciding on which model to use, evaluate your dataset's characteristics carefully:

Prophet is particularly beneficial when your data includes strong seasonal effects and you have some domain knowledge to incorporate holidays and other special events.
ARIMA works well if your time series is relatively stable and exhibits trends and autocorrelation that ARIMA's structure can model.
LSTMs offer the greatest flexibility and learning capability for complex and long-term dependencies, but at the cost of needing more data and computational resources.

Each model has its strengths, and sometimes a combination or ensemble approach might yield the best results. Trial and experimentation with each model on your specific dataset are essential steps to identify the most suitable model for your forecasting goals.

from prophet import Prophet from pyspark.sql.functions import col, to_date # Prophet requires at the minimum 2 columns - ds & y train_df = no_outliers_df.select(to_date(col("<date>")).alias("ds"), col("<metric>"), col("<metric_value>").alias("y").cast("double")).orderBy(col("ds").desc()) # set model parameters prophet_model = Prophet( interval_width=0.95, growth='linear', daily_seasonality=True, weekly_seasonality=True, yearly_seasonality=True, seasonality_mode='additive' ) # fit the model to historical data history_pd = train_df.toPandas() prophet_model.fit(history_pd)

Fit Data & Build Forecast

To effectively utilize Prophet for forecasting in Databricks, follow these concise steps:

Data Preparation: Your dataset must have two columns: one for the datetime (ds) in YYYY-MM-DD format, and another for the metric you're forecasting (y). This format is crucial for the model to correctly learn from your data.
Model Initialization and Fitting: Once your data is properly formatted, you will initiate a Prophet model instance and fit it with your historical data. During this step, Prophet analyzes your data's patterns to prepare for forecasting.
Forecasting Future Values: After fitting the model, you will create a DataFrame that outlines the future dates you wish to predict. Using this DataFrame, Prophet will generate forecasts for the specified future dates.

#Fit the model to historical data history_pd = train_df.toPandas() prophet_model.fit(history_pd) #Define Dataset with historical dates & 10-days beyond the last available date future_pd = prophet_model.make_future_dataframe( periods=10, freq='d', include_history=True ) #Forecast forecast_pd = prophet_model.predict(future_pd) display(forecast_pd)

Evaluation

This can be done by comparing the model’s forecasts to actual data and calculating performance metrics like Mean Squared Error (MSE). Performance Metrics:

Mean Squared Error (MSE): Measures the average squared difference between the estimated values and the actual value, offering a view of the overall variance in the forecasting errors. Lower MSE values denote a model with fewer errors.
Root Mean Squared Error (RMSE): Represents the square root of MSE, thus re-scaling errors to the original units of the target variable, which improves interpretability.
Mean Absolute Error (MAE): Averages the absolute differences between predicted and actual values. Unlike MSE, MAE is more robust to outliers, as it does not square the errors.

Interpreting the Metrics

MSE and RMSE are more sensitive to outliers due to squaring the errors, often used when large errors are particularly undesirable.
MAE is straightforward and easy to interpret, as it directly represents the average error.

The choice between these metrics should be informed by your specific forecasting objectives and the nature of the data. In practice, assessing model performance might involve looking at multiple metrics to get a comprehensive view of the model's accuracy.

import pandas as pd from sklearn.metrics import mean_squared_error, mean_absolute_error, r2_score from math import sqrt from datetime import date # get historical actuals & predictions for comparison actuals_pd = history_pd[history_pd['ds'] < date(2024, 4, 29)]['y'] #Update it to max date on your dataset predicted_pd = forecast_pd[forecast_pd['ds'] < pd.to_datetime('2024-04-29')]['yhat'] #Update it to max date on your dataset # calculate evaluation metrics mae = mean_absolute_error(actuals_pd, predicted_pd) mse = mean_squared_error(actuals_pd, predicted_pd) rmse = sqrt(mse) # Print other metrics print(f"MAE: {mae}") print(f"MSE: {mse}") print(f"RMSE: {rmse}")

Log the model with MLflow

Model Logging is critical for tracking the performance and changes to models over time, ensuring reproducibility and accountability. Techniques include:

MLflow Logging: Utilize MLflow's robust platform for logging models, parameters, and artifacts. It supports structured experiment tracking, perfect for recording and comparing different versions of your models.
Custom Logging: Implement tailored logging approaches to capture unique model insights or additional metadata not standardly logged by existing tools.

Benefits of Logging with MLflow

Logging models with MLflow offers several advantages:

Reproducibility: By capturing all necessary details of the experimentation phase, MLflow makes it easier to replicate results and understand decision-making processes.
Model Registry: MLflow allows for versioning of models, making it simple to manage and deploy specific model versions based on performance metrics.
Collaboration and Sharing: Teams can leverage MLflow’s centralized model storage to share models and results, enhancing collaboration.

For the purpose of this tutorial, we demonstrate how to efficiently log a Prophet model using MLflow, capturing essential information that supports further analysis and model deployment.

import mlflow from mlflow.tracking import MlflowClient from mlflow.models.signature import infer_signature from mlflow.pyfunc import PythonModel, log_model # Define the catalog, schema, and model name for organizing the model within the MLflow model registry catalog = "catalog_name" #Update it to your catalog name schema = "schema_name" #Update it to your schema name model_name = "forecastingmodel" #Update it to your model name class MyPythonModel(mlflow.pyfunc.PythonModel): def __init__(self, model): self.model = model def predict(self, context, model_input): future_pd = self.model.make_future_dataframe(periods=10, freq="d", include_history=True) forecast_pd = self.model.predict(future_pd) return forecast_pd[["ds", "yhat", "yhat_upper", "yhat_lower"]] wrapped_model = MyPythonModel(prophet_model) # Enable MLflow auto logging for tracking machine learning metrics and artifacts with mlflow.start_run(run_name="Prophet Model Run") as run: input_example = history_pd.head()[["ds", "y"]] output_example = prophet_model.predict(input_example).iloc[:10] # Log calculated metrics mlflow.log_metric("mae", mae) mlflow.log_metric("rmse", rmse) print(output_example) # Infer the signature of the machine learning model signature = infer_signature(input_example, output_example) # Update dependencies in the default conda environment env = mlflow.pyfunc.get_default_conda_env() env['dependencies'][-1]['pip'] += ["prophet==1.1.5"] env['dependencies'][-1]['pip'] += ["pandas==1.5.3"] env['dependencies'][-1]['pip'] += ["pyspark==3.5.1"] env['dependencies'][-1]['pip'] += ["grpcio==1.62.0"] env['dependencies'][-1]['pip'] += ["grpcio_status==1.62.0"] # Log the trained model to MLflow with the inferred signature model_log = log_model( artifact_path="forecasting_model", python_model=wrapped_model, signature=signature, input_example=input_example, registered_model_name=f"{catalog}.{schema}.{model_name}", conda_env=env ) # Retain the "run_id" for use with other MLflow functionalities like registering the model run_id = run.info.run_uuid

#Get the latest Model Version def get_latest_model_version(model_name:str = None): latest_version = 1 mlflow_client = MlflowClient() for mv in mlflow_client.search_model_versions(f"name='{model_name}'"): version_int = int(mv.version) if version_int > latest_version: latest_version = version_int return latest_version model_version = get_latest_model_version(f"{catalog}.{schema}.{model_name}")

Deploy Models on Databricks

Deploying machine learning models into production on Databricks can be achieved through two primary methods: MLflow for batch inference and prediction, and Databricks Model Serving for real-time inference. Each serves different use cases based on the requirement for real-time responses and the scale of data processing.

MLflow for Batch Inference and Prediction: Batch processing is ideal for scenarios where predictions can be made on large datasets at once without the need for immediate responses. This method fits well with scheduled analytics and reporting.
Databricks Model Serving for Real-Time Inference: This method is better suited for scenarios where low latency and real-time responses are important.

import mlflow, os import requests, json from databricks.sdk import WorkspaceClient from databricks.sdk.service.serving import EndpointCoreConfigInput, ServedModelInput from mlflow.deployments import get_deploy_client serving_endpoint_name = "forecasting_model_serving" # Get the API endpoint and token for the current notebook context API_ROOT = dbutils.notebook.entry_point.getDbutils().notebook().getContext().apiUrl().get() API_TOKEN = dbutils.notebook.entry_point.getDbutils().notebook().getContext().apiToken().getOrElse(None) client = get_deploy_client("databricks") # Check if the endpoint already exists existing_endpoint = next( (e for e in client.list_endpoints() if e['name'] == serving_endpoint_name), None ) # Update the endpoint configuration endpoint_config = { "served_entities": [ { "entity_name": f"{catalog}.{schema}.{model_name}", "entity_version": model_version, "workload_size": "Small", "workload_type": "CPU", "scale_to_zero_enabled": True } ] } if existing_endpoint is not None: # Update the existing endpoint endpoint = client.update_endpoint( endpoint=serving_endpoint_name, config=endpoint_config ) else: # Create a new endpoint if it does not exist endpoint = client.create_endpoint( name=serving_endpoint_name, config=endpoint_config )

# Wait for Endpoint to be ready import time from datetime import datetime, timedelta # Define the maximum wait time (20 minutes) max_wait_time = timedelta(minutes=20) deadline = datetime.now() + max_wait_time # Function to check the status of the endpoint def check_endpoint_status(client, endpoint_name): endpoints = client.list_endpoints() for endpoint in endpoints: if endpoint['name'] == endpoint_name: return endpoint return None # Wait for the endpoint to be ready or until the deadline is reached while datetime.now() < deadline: endpoint_info = check_endpoint_status(client, serving_endpoint_name) if endpoint_info is not None and str(endpoint_info['state']['ready']).lower() == 'ready' and str(endpoint_info['state']['config_update']).lower() != 'in_progress': print(f"Endpoint {serving_endpoint_name} is ready.") break else: print(f"Waiting for endpoint {serving_endpoint_name} to be ready. Current status: {endpoint_info['state'] if endpoint_info else 'Not Found'}") time.sleep(60) # Wait for 60 seconds before checking again else: print(f"Timeout reached. Endpoint {serving_endpoint_name} may not be ready.") displayHTML( f'Your Model Endpoint Serving is now available. Open the <a href="/ml/endpoints/{serving_endpoint_name}">Model Serving Endpoint page</a> for more details.' )

Build Forecast and Continuous Improvement

After developing and deploying your machine learning model, the final step is to utilize the model to make predictions. This process involves sending new data to the model endpoint and interpreting the predictions returned by the model.

Generate Forecasts: Use your model to predict future values based on historical data.
Validate and Iterate: Continuously validate your model against new data and iterate to improve accuracy and reliability.

#Predict using Served Model import requests import json from datetime import date, datetime # Custom encoder for handling date and datetime objects in JSON serialization class CustomJSONEncoder(json.JSONEncoder): def default(self, obj): if isinstance(obj, (datetime, date)): return obj.isoformat() return json.JSONEncoder.default(self, obj) # Prepare data payload from DataFrame for model invocation data_payload = {"dataframe_records": history_pd.to_dict(orient='records')} data_json = json.dumps(data_payload, cls=CustomJSONEncoder) # Setup headers for the POST request headers = { "Content-Type": "application/json", "Authorization": f"Bearer {API_TOKEN}", } # Endpoint URL for model invocation serving_endpoint_url = f"{API_ROOT}/serving-endpoints/{serving_endpoint_name}/invocations" # API call to deploy model and obtain predictions response = requests.post(serving_endpoint_url, headers=headers, data=data_json) # Check and display the response if response.status_code == 200: predictions = response.json() print("Predictions:", predictions) else: print("Failed to make predictions")

#Visualize the Predictions import pandas as pd import matplotlib.pyplot as plt from datetime import datetime, timedelta # Convert predictions JSON to DataFrame pred_df = pd.json_normalize(predictions['predictions']) # Ensure 'ds' columns are datetime objects for merging and filtering history_pd['ds'] = pd.to_datetime(history_pd['ds']) pred_df['ds'] = pd.to_datetime(pred_df['ds']) # Merge historical and prediction data on 'ds' combined_df = pd.merge(left=pred_df, right=history_pd, on='ds', how='left') # Filter for data from the last 60 days combined_df = combined_df[combined_df['ds'] >= history_pd['ds'].max() - timedelta(days=60)] # Plotting setup plt.figure(figsize=(12, 6)) # Plot actual values and predictions plt.plot(combined_df['ds'], combined_df['y'], label='Actual', color='black') plt.plot(combined_df['ds'], combined_df['yhat'], label='Predicted', color='blue') # Indicate prediction uncertainty plt.fill_between(combined_df['ds'], combined_df['yhat_lower'], combined_df['yhat_upper'], color='gray', alpha=0.2) # Finalize plot plt.title('Model Predictions vs Actual Values') plt.xlabel('Date') plt.ylabel('Value') plt.legend() plt.grid(True) # Display the plot plt.show()

In conclusion, successfully forecasting on Databricks hinges on a thorough understanding of your data, meticulous preparation, strategic model selection, and continuous improvement through iteration, logging, and serving. By integrating these practices into your workflow, you can develop powerful forecasting models that drive informed decisions for your organization.

Getting Started with Databricks - Build a simple Lakehouse analytics pipeline

DatabricksGuide — Thu, 15 Aug 2024 18:09:35 GMT

Getting started with Databricks - Build a simple Lakehouse analytics pipeline

The demo example in this guide illustrates a lakehouse analytics pipeline using the well-known NYC taxi trip dataset. This public dataset is also available at Kaggle.

This guide provides a practical demonstration of how to structure a data pipeline using the medallion architecture, progressively refining and analyzing data as it moves through each layer. It also demonstrates the pipeline’s integration with Unity Catalog, showing how to set up data lineage and governance. Finally, it covers AI/BI dashboards for enhancing analytical decision-making and sharing insights with team members.

This is a beginner’s tutorial with hands-on instructions to execute in your own Databricks workspace. You can request a free 14-day trial.

Understanding the medallion architecture

The medallion architecture is a method for organizing and refining data in a lakehouse by moving it through three layers—Bronze (raw data), Silver (cleaned data), and Gold (final, ready-to-use data).

Step 1: Create a notebook and add SQL pipeline code

In your Databricks workspace, click "+New" in the left sidebar and select Notebook. Name the notebook “NYTaxi Pipeline SQL.”
Set the notebook’s default language to SQL next to its name. We want to code the pipeline in SQL for simplicity.
Copy and paste the following code into your new SQL notebook.

Bronze layer

Bronze Table: Raw data ingestion
Here, raw taxi trip data is ingested, with a basic data quality check applied to ensure trip distances are positive.

- Bronze layer: Raw data ingestion CREATE OR REPLACE TABLE taxi_raw_records AS SELECT * FROM samples.nyctaxi.trips WHERE trip_distance > 0.0;

-Silver layer

The Silver layer creates two tables:
Silver Table 1: Flagged rides
This table identifies potentially suspicious rides based on fare and distance criteria.

-- Silver Table 1: Flagged rides CREATE OR REPLACE TABLE flagged_rides AS SELECT date_trunc("week", tpep_pickup_datetime) AS week, pickup_zip AS zip, fare_amount, trip_distance FROM taxi_raw_records WHERE ((pickup_zip = dropoff_zip AND fare_amount > 50) OR (trip_distance < 5 AND fare_amount > 50));

Silver Table 2: Weekly statistics
This silver table calculates weekly average fares and trip distances.

- Silver layer 2: Weekly statistics CREATE OR REPLACE TABLE weekly_stats AS SELECT date_trunc("week", tpep_pickup_datetime) AS week, AVG(fare_amount) AS avg_amount, AVG(trip_distance) AS avg_distance FROM taxi_raw_records GROUP BY week ORDER BY week ASC;

Gold layer
Gold Table 1: Top N rides
Here, these silver tables are integrated to provide a comprehensive view of the top three highest-fare rides.

-- Gold layer: Top N rides to investigate CREATE OR REPLACE TABLE top_n AS SELECT ws.week, ROUND(ws.avg_amount, 2) AS avg_amount, ROUND(ws.avg_distance, 3) AS avg_distance, fr.fare_amount, fr.trip_distance, fr.zip FROM flagged_rides fr LEFT JOIN weekly_stats ws ON ws.week = fr.week ORDER BY fr.fare_amount DESC LIMIT 3;

Step 2: Schedule a notebook job

To ensure that the Bronze, Silver, and Gold tables are regularly updated with fresh data, it’s recommended to schedule the notebook as a job to run periodically. This will keep the tables current with the latest information. To schedule a notebook job to run periodically:

In the notebook, click the Schedule button at the top right.
In the Schedule dialog, optionally enter a name for the job. The default name is the name of the notebook.
Scheduled runs allow you to define a schedule for your job run. Adjust the frequency, time, and time zone for the job run.
Leave everything else as is and click Create.
Once the schedule is successfully created, click Run Now to trigger a job run for the NYCTaxiSQL Pipeline.

Step 3: Discover data using Catalog Explorer

Explore and manage your generated datasets through Unity Catalog using Catalog Explorer. Unity Catalog organizes data in a three-level namespace: Catalog.Schema.Table. Follow the steps below to examine the data produced by your pipeline and visualize its lineage. This allows you to inspect sample data, view table details, and explore the end-to-end data lineage of your pipeline objects.

Click Catalog in the sidebar to open the Catalog Explorer.
The newly created tables from our NYCTaxiSQLPipeline should be accessible by clicking the Recents top on the top of the page.
Click weekly_stats to load the table details.
Click Lineage and then Lineage Graph to view a graphical representation of all the upstream and downstream tables from weekly_stats.

Step 4: Create a Dashboard

Use the Create button on the right side to create a Dashboard from the weekly_stats table. This automatically creates a new dashboard for the weekly_stats table. Rename the dashboard on the top as NYCTaxiPipelineDash.
Start building out your dashboard by using the Ask the Assistant prompt at the top of the dashboard to create your first chart using AI. Click one of the auto-generated prompts to get started.
Add more charts by using the “Add a Visualization” widget from the selector at the bottom and place the cursor on a space on the dashboard. Here are some additional prompts:

Show a scatter plot of the trip distance and average fare amount by day of the week.
Visualize total trips across zip codes as a bar chart

Click Accept to save the AI-generated chart or use the widget on the right to regenerate the chart. Use thebutton on the upper-right of each chart to provide a different prompt to update the chart.
Alternatively, use the chart builder on the right side to choose a chart type and then select values for the x-axis and y-axis.

Step 5: Publish and distribute your dashboard

While you develop a dashboard, your progress is saved as a draft. To create a clean copy for easy consumption, publish your dashboard. After you publish a dashboard, the published version remains intact until you publish again, even if you make changes to the draft. You can make modifications and improvements to the draft version without affecting the published copy.

When you’re done adding charts to this dashboard, click Publish in the upper-right corner to create a clean copy of the current dashboard.
Click Publish. Your dashboard is now ready to be shared with other team members.

Add users to your Databricks Workspace

In the top bar of the Databricks workspace, click your username and then click Settings.
In the sidebar, click Identity and Access.
Next to Users, click Manage.
Click Add user, and then click Add new.
Enter the user’s email address, and then click Add.

Continue to add as many users to your account as you would like. New users receive an email prompting them to set up their account.

Share the dashboard with colleagues

To manage access to the dashboard, click Share at the top of the dashboard to open the permissions dialog.
Enter the email address of the user you want to share the dashboard with and click "Add." They will receive an email notification and will be able to access the dashboard in Databricks.
[Optional] To share the dashboard with all colleagues, add the “All Users” group to the dashboard’s access list with “Can View” or “Can Run” permissions. You can also copy the dashboard’s URL to your clipboard using the “Copy link” button and send it directly to your colleagues. 🎉

Congratulations!

You’ve now created, run, and analyzed an analytics pipeline demonstrating data processing, data quality checks, and the creation of analytics-ready tables and dashboards. Not only have you gained insights into the taxi trip data, but you’ve also likely learned how expensive a ride in the Big Apple can be if things go wrong — perhaps it’s time to consider the subway!

As you become more comfortable with ETL pipelines, you can expand this pipeline with more complex transformations, data streaming, and more comprehensive data quality checks.

To learn more about Databricks Jobs, see What is Databricks Jobs?
To learn more about Delta Lake, see What is Delta Lake?
To learn more about data processing pipelines with Delta Live Tables, see What is Delta Live Tables?

Getting Started with Databricks SQL Serverless on Azure

DatabricksGuide — Mon, 11 Mar 2024 16:28:40 GMT

Getting Started with Databricks SQL Serverless on Azure

This is an Azure admin guide for existing Databricks SQL customers interested in SQL Serverless features. This guide covers the following topics:

What is Serverless Architecture
Security on Serverless Architecture
Why use Databricks SQL Serverless
Serverless SQL Setup Steps (in 5 minutes)
Additional Resources (troubleshooting steps, customer FAQs)

To learn more about Databricks SQL and data warehousing on the Databricks Data Intelligence Platform, visit our website.

What is Serverless Architecture?

Serverless is an architectural design where the compute layer and its associated resources are managed in the Databricks account rather than your Azure account. That means your resource and infrastructure management shifts to Databricks. A serverless architecture is advantageous in the following ways:

Productivity: cloud resources are efficiently managed by Databricks, eliminating management overhead and providing instant compute that boosts user productivity.

Efficiency: Serverless ensures near-instant start-up/scaling time while preventing underutilization. You benefit by only paying for the used compute and eliminating unnecessary idle time.
Reliability: Serverless is fully managed with capacity, security, patching and upgrades managed automatically, which eliminates concerns about security policies and reliability issues such as capacity shortages or quota limitations.

To learn more, please visit our documentation for more details on Databricks serverless compute.

Security on Serverless Architecture

To safeguard customer data, serverless workloads run within multiple isolation layers. These security layers form the foundation of Databricks’ commitment to providing a secure and reliable environment for even the most sensitive workloads. They include but are not limited to:

Dedicated compute resources
Network segmentation
Encryption at rest and in transit
Principle of least privilege

The high-level connections into and out of your Azure Databricks account for serverless is shown below:

To learn more about security with serverless compute, visit our website.

Why use Databricks SQL Serverless?

Databricks SQL is best with Serverless. Here are three main benefits of Serverless over Pro and Classic warehouses:

Instant and elastic compute: Serverless removes the need to wait for infrastructure resources to run queries or over provision resources to handle spikes in usage. Serverless warehouses dynamically handle scaling using intelligent workload management.
Zero Management: Serverless warehouses are fully managed by Databricks, eliminating the burden of capacity management, patching, upgrading and performance optimization. This leads to a simple and predictable pricing model compared to Pro or Classic.
Lower TCO: Serverless warehouses automatically provision and scale resources right when you need them. This avoids over-provisioning and reduces idle time, which can help reduce your TCO (Total Cost of Ownership).

To learn more about Serverless advantages or specific projected cost-savings scenarios from migrating Classic/Pro warehouses to Serverless, visit our pricing page. For more help, contact onboarding-help@databricks.com or your Databricks account team directly.

SQL Serverless Setup Steps [5 minute guide]:

Step 0: Check the required prerequisites

This section of the guide assumes you have the following:

Workspace admin permissions on your Databricks account and intended workspace.
Admin permissions to your Azure account.
Non-free trial Databricks account. If you are interested in SQL Serverless on a free trial, contact onboarding-help@databricks.com.
Premium or enterprise account subscription. This can be verified in your Databricks account console. Below is an example Databricks account on the correct premium plan. If you are on standard, upgrade to premium or contact onboarding-help@databricks.com for help.
Intended workspace is in a supported region for Serverless warehouses.
Intended workspace is on the Premium plan
Intended workspace does not use an external Apache Hive metastore. SQL Serverless does not support this today, and we require upgrading to Unity Catalog. For help, contact your Databricks account team or onboarding-help@databricks.com.

Step 1: Activate and Configure Serverless

Serverless is automatically enabled by default in Databricks SQL. If your account uses Azure Private Link, Azure Storage firewall, or NCC private end points, visit the Appendix section for additional manual setup steps.

Step 2: Create a serverless warehouse and grant permissions

Log into your workspace and click on SQL Warehouses on the left sidebar. Use this guided tour for a step by step walkthrough in a demo workspace. Alternatively, visit our documentation for setup instructions.

Step 3: Convert all existing Pro and Classic Warehouses to Serverless

Log into your workspace and click on SQL Warehouses on the left sidebar. To upgrade a SQL warehouse to serverless, click the kebab menu, then click Upgrade to Serverless.

Step 4: Enable your users on Serverless

Your serverless setup on Azure is now complete! Enable your users to start using serverless for their SQL workloads. Ensure your serverless warehouse(s) are properly configured with user permissions. Visit our documentation for more details. Workspace users are able to use warehouses in multiple ways including:

Additional Resources

Troubleshooting Steps

Scenario

Action Required

If you use a Azure storage firewall

Configure your firewall to allow access from serverless compute nodes. This will allow Azure Databricks serverless subnet to contact your Azure storage. Visit our documentation for setup steps. Completing this will prevent unauthorized access and added security of your data while using serverless computing on Azure Databricks.

If you use Azure private link

Azure private link is in gated public preview. Please reach out to onboarding-help@databricks.com or contact your Databricks account team for more information on how to enroll. Azure Private Link support will be generally available in April 2024, please visit our blog announcement for more information.

If you use private connectivity for serverless compute

Configure your NCCs in the account console and attach them to one or more workspaces. This allows Azure Databricks to create a private endpoint request to your Azure resource. Below is a example of correctly configured end point rules for serverless compute in the Azure portal.

This is in public preview, contact onboarding-help@databricks.com or your Databricks account team for additional help

Customer FAQs

#	Question	Answer
1	Does SQL Serverless require Unity Catalog to work?	No, you can start using Serverless SQL today and get started on your current Hive Metastore setup. You can transition to Unity Catalog while actively using Serverless SQL. To upgrade your workspace to Unity Catalog, follow our tutorial documentation or contact your Databricks account team.
3	Does SQL Serverless support HIPAA?	Yes, we support the regions listed here. To prioritize a specific region for HIPPA support, contact onboarding-help@databricks.com or your Databricks account team directly.
4	Do SQL warehouses support compliance security profiles?	Yes, but it varies by region. See the regions with support for compliance security profiles here. To prioritize your account’s enrollment, contact onboarding-help@databricks.com or your Databricks account team directly.
5	Will using SQL warehouses affect my runtime clusters?	No, using serverless warehouses affects only the workloads using Databricks SQL. Databricks Runtime clusters continue to work with notebooks and jobs. Databricks Runtime clusters always run in the classic compute plane in your Azure account.
6	How can I migrate to SQL warehouses if I provision my warehouses using IaC tools like Terraform?	Upgrading to SQL Serverless is a simple code change in your IaC scripts. Contact your account team if you need additional support.
7	How do I find the cost of running a SQL Query against SQL Serverless? I want to see the DBU usage from a specific SQL query.	This can be done using system tables on Databricks SQL. Note, this requires a Unity Catalog enabled workspace. If you need to upgrade your workspace to Unity Catalog, follow our documentation or contact your Databricks account team.

Learn More

Register for our weekly Hands-on Workshop: Databricks SQL and Serverless
Register for free live onboarding training for Databricks SQL
Watch the Data + AI Summit talk on how SQL Serverless uses Machine Learning to drive best price to performance.
If you encounter any errors or need additional help, email us at onboarding-help@databricks.com for assistance. For specific questions and serverless POCs, contact your Databricks account team.

Getting Started with Databricks SQL Serverless on AWS

DatabricksGuide — Sat, 09 Mar 2024 16:53:09 GMT

Getting Started with Databricks SQL Serverless on AWS

This is an AWS admin guide for existing Databricks SQL customers interested in SQL Serverless features. This guide covers the following topics:

What is Serverless Architecture
Security on Serverless Architecture
Why use Databricks SQL Serverless
Serverless SQL Setup Steps (in 5 minutes)
Additional Resources (troubleshooting steps, customer FAQs)

To learn more about Databricks SQL and data warehousing on the Databricks Intelligent Data Platform, visit our website.

What is Serverless Architecture?

Serverless is an architectural design where the compute layer and its associated resources are managed in the Databricks account rather than your AWS account. That means your resource and infrastructure management shifts to Databricks. A serverless architecture is advantageous in the following ways:

Productivity: cloud resources are efficiently managed by Databricks, eliminating management overhead and providing instant compute that boosts user productivity.

Efficiency: Serverless ensures near-instant start-up/scaling time while preventing underutilization. You benefit by only paying for the used compute and eliminating unnecessary idle time.
Reliability: Serverless is fully managed with capacity, security, patching and upgrades managed automatically, which eliminates concerns about security policies and reliability issues such as capacity shortages or quota limitations.

To learn more, please visit our documentation for more details on Databricks serverless compute.

Security on Serverless Architecture

Dedicated compute resources
Network segmentation
Encryption at rest and in transit
Principle of least privilege

To learn more about security with serverless compute, visit our website.

Why use Databricks SQL Serverless?

Databricks SQL is best with Serverless. Here are three main benefits of Serverless over Pro and Classic warehouses:

Instant and elastic compute: Serverless removes the need to wait for infrastructure resources to run queries or over provision resources to handle spikes in usage. Serverless warehouses dynamically handle scaling using intelligent workload management.
Zero Management: Serverless warehouses are fully managed by Databricks, eliminating the burden of capacity management, patching, upgrading and performance optimization. This leads to a simple and predictable pricing model compared to Pro or Classic.
Lower TCO: Serverless warehouses automatically provision and scale resources right when you need them. This avoids over-provisioning and reduces idle time, which can help reduce your TCO (Total Cost of Ownership).

SQL Serverless Setup Steps [5 minute guide]:

Step 0: Check the required prerequisites

This section of the guide assumes you have the following:

Workspace admin permissions on your Databricks account and intended workspace.
Permissions access to your AWS account.
Non-free trial Databricks account. If you are interested in SQLServerless on a free trial, contact onboarding-help@databricks.com.
Premium or enterprise account subscription. This can be verified in your Databricks account console. Below is an example Databricks account on the correct premium plan. If you are on standard, upgrade to premium or contact onboarding-help@databricks.com for help.
Intended workspace is in a supported region for Serverless warehouses.
Intended workspace does not use an external Apache Hive metastore. SQL Serverless does not support this today, and we require upgrading to Unity Catalog. For help, contact your Databricks account team or onboarding-help@databricks.com.

Step 1: Activate Serverless

Serverless is automatically enabled by default in Databricks SQL if your account was created recently. If your account was created prior to 6/24/2022, check the troubleshooting steps in the Appendix for the manual activation steps to enable serverless. This includes manually accepting your ToS and configuring your AWS instance profile.

Step 2: Create a serverless warehouse and grant permissions

Step 3: Convert all existing Pro and Classic Warehouses to Serverless

Log into your workspace and click on SQL Warehouses on the left sidebar. Follow two step instructions in our documentation.

Step 4: Enable your users on serverless

Your serverless setup on AWS is now complete! Enable your users to start using serverless for their SQL workloads. Ensure your serverless warehouse(s) are properly configured with user permissions. Visit our documentation for more details. Workspace users are able to use warehouses in multiple ways including:

Additional Resources

Troubleshooting Steps

Scenario

Action Required

If your Databricks account was created before 3/28/2022

Verify that you have manually accepted the updated terms of use for serverless compute. Visit our documentation for setup steps. Below is an example of a account that is correctly enabled for Serverless compute.

If your Databricks account was created before 6/24/2022

Validate that your AWS instance profile supports Serverless SQL warehouses. This is required because Databricks is an external IAM role. If your AWS instance profile was created after this date, it most likely has the trust relationship statement created already using AWS quickstart or manually.

1. Follow our documentation for the setup steps to update the trust relationship statement to support serverless SQL warehouses. Below is an example of a correctly updated IAM role in the AWS console.

2. If your role ARN and Instance Profile ARNs do not match, first follow the required steps as listed in the documentation here to update your role ARN.

Customer FAQs

#	Question	Answer
1	Does SQL Serverless require Unity Catalog to work?	No, you can start using Serverless SQL today and get started on your current Hive Metastore setup. You can transition to Unity Catalog while actively using Serverless SQL. To upgrade your workspace to Unity Catalog, follow our tutorial documentation or contact your Databricks account team.
2	Does SQLServerless support AWS Glue?	Yes, AWS Glue can be used as the workspace legacy metastore if you already have Glue set up for your SQL workloads. No additional configuration is needed.
3	Does SQL Serverless support HIPAA?	Yes, we support the regions listed here. To prioritize a specific region for HIPPA support, contact onboarding-help@databricks.com or your Databricks account team directly.
4	Do SQL warehouses support compliance security profiles?	Yes, but it varies by region. See the regions with support for compliance security profiles here. To prioritize your account’s enrollment, contact onboarding-help@databricks.com or your Databricks account team directly.
5	Will using SQL warehouses affect my runtime clusters?	No, using serverless warehouses affects only the workloads using Databricks SQL. Databricks Runtime clusters continue to work with notebooks and jobs. Databricks Runtime clusters always run in the classic compute plane in your AWS account.
6	How can I migrate to SQL warehouses if I provision my warehouses using IaC tools like Terraform?	Upgrading to SQL Serverless is a simple code change in your IaC scripts. Contact your account team if you need additional support.
7	How do I find the cost of running a SQL Query against SQL Serverless? I want to see the DBU usage from a specific SQL query.	This can be done using system tables on Databricks SQL. Note, this requires a Unity Catalog enabled workspace. If you need to upgrade your workspace to Unity Catalog, follow our documentation or contact your Databricks account team.

Learn More

Register for our weekly Hands-on Workshop: Databricks SQL and Serverless
Register for free live onboarding training for Databricks SQL
Watch the Data + AI Summit talk on how SQL Serverless uses Machine Learning to drive best price to performance.
If you encounter any errors or need additional help, email us at onboarding-help@databricks.com for assistance. For specific questions and serverless POCs, contact your Databricks account team.