Databricks Community

sonakshipandey

Integrating your data warehouse with a modern business intelligence (BI) platform is vital in today's data-centric landscape. Connecting a user-friendly tool like Sigma to Databricks SQL warehouse unlocks fast, scalable analytics on even the largest datasets.

This guide details the secure, recommended way to integrate Sigma and Databricks: using OAuth 2.0 for authentication. We'll delve into the advantages of OAuth over Basic Auth and provide a detailed, step-by-step process for connecting Sigma with Databricks.

What is Sigma?

Sigma is a cloud-native analytics platform that provides a familiar spreadsheet-like user interface. It enables you to query, profile, visualize, and explore massive datasets stored and shared in leading data platforms like Databricks. Sigma directly connects to Databricks and a proprietary SQL-generation engine that translates user interactions on a familiar spreadsheet interface into machine-optimized SQL. Sigma unlocks the power of the Databricks Lakehouse Platform by providing speed, scalability, and security with the flexibility to pivot billions of rows of data.

What is Databricks SQL?

Databricks SQL is a serverless AI-powered data warehouse built on the Databricks Platform. It brings a familiar SQL analytics experience to your lakehouse data and supports warehousing and BI workloads.

Key features include:

Offers a fully managed, serverless experience with no cluster management required.
Seamlessly integrates with many major BI tools like Sigma via optimized connectors.
Unified governance via Unity Catalog for fine-grained security and access controls.

Databricks SQL delivers the performance and usability of a traditional data warehouse, with the flexibility and economics of a data lake.

The Advantages of Connecting Sigma to Databricks

Connecting Sigma with Databricks SQL brings the power of lakehouse to BI tools:

Drag-and-drop interface: The user interface is designed to be intuitive and user-friendly. Users can drag and drop data fields to create charts, tables, and graphs without writing complex code or queries.
Real-time data analysis: Users can analyze data in real time. This means that as they make changes to their analysis, the results are reflected in real time, allowing for quick and agile decision-making.
Collaboration features: Users can now collaborate across teams in real time, sharing worksheets, dashboards, comments, and chats.
Visualizations: Users can create a range of visualizations with just a few clicks, making it easy to communicate insights to stakeholders.
Granular Access Controls: Sigma honors Databricks’ Unity Catalog controls, Which Include All the fine-grained access controls, including row and column-level filtering.

Connecting Sigma and Databricks

When connecting Sigma to Databricks, you have a few authentication options. Understanding the differences is crucial to making the right choice for your organization.

Authentication Methods

Basic Auth (Personal Access Token): This method uses a Databricks Personal Access Token (PAT) for authentication. You generate a long-lived token in Databricks and paste it into the Sigma connection settings. While functional, it presents security and management challenges.
OAuth (Recommended): This is a modern, token-based authentication protocol. Instead of storing credentials, it uses a secure handshake, where Databricks grants Sigma temporary access tokens per user. Users log in with their standard Databricks credentials (often via Single Sign-On), and Sigma never handles or stores their passwords.

Note: Sigma does not support connecting via a Databricks Service Principal OAuth.

Why OAuth Integration is the Better Choice

While setting up Basic Auth with Personal Access Tokens (PAT) is simple, it limits the connection to one Databricks user or a service principal, and Sigma users are limited by this user’s access and permissions.

OAuth is the recommended method for any enterprise environment as it provides several advantages:

Enhanced Security: With OAuth, sensitive credentials like passwords or long-lived tokens are never stored in the BI tool. Access tokens are short-lived and can be easily revoked, minimizing the risk associated with credential exposure.
Centralized User Management: OAuth leverages your existing Databricks user identities. When an employee leaves the company and their Databricks account is deactivated, their access is automatically revoked in Sigma. With PATs, you must manually track and delete the specific token, which adds operational burden.
Seamless User Experience: Users get a true Single Sign-On (SSO) experience. They simply click "Log in with Databricks" and are authenticated through your company's existing identity provider without managing a separate set of credentials or tokens.

Step-by-Step Connection Guide

Let's walk through setting up the OAuth connection between Databricks and Sigma. It is a two-part process: first, you'll configure an OAuth application in Databricks, and then use those details to create the connection in Sigma.

Prerequisite:

The Requirements section of the Sigma documentation provides a comprehensive list of the permissions needed on both the Sigma and Databricks sides. Ensure this is set up before you begin following the steps below.
We assume that you have already created a Databricks SQL Warehouse. If not, see the Databricks documentation for instructions on how to set one up.

Part 1: Configure the OAuth Application in Databricks

You must be a Databricks account administrator to perform these steps.

Log in to your Databricks Account Console.
In the left sidebar, navigate to Settings and click App Connections tab.
Click the Add Connection button in the top right.
Fill in the application details:

Application Name: Give it a descriptive name, like Sigma Connection.
Redirect URLs: Paste the following URL provided by Sigma: https://aws-api.sigmacomputing.com/api/v2/oauth/1/authcode
- This is the URL where Databricks will send users (and the OAuth authorization code) after they approve the connection.
Access Scopes: Grant the application permission to All APIs
Client secret: Check. Generate a client secret.
Choose a custom Access token TTL and Refresh token TTL in minutes, or leave it to the default.

Click Add.

Important: Databricks will now display the Client ID (the unique ID for Sigma app) and Client Secret (confidential password used by Sigma to prove its identity). Copy both of these values immediately and store them securely. The Client Secret will not be shown again after you close this window.

This is what your App Connections should look like :

Part 2: Create the Connection in Sigma

Now, with your Client ID and Secret in hand, head over to Sigma. You'll need to be a Sigma Admin.

Log in to your Sigma instance.
Click your user icon in the top right and select Administration.
Go to the Connections page and click the Create Connection button.
For the Connection Details

Specify the connection Name like Databricks Sigma
Select Databricks as the connection Type.

To fill in Connection Credentials, we will use the Databricks SQL Warehouse information from the prerequisites.
- Host: Enter your Databricks Server hostname (as seen in the picture above).
- HTTP path: Enter the HTTP path from your Databricks SQL warehouse.
- Authentication: Select OAuth.
Configure OAuth Features:
- Scopes: Add any additional OAuth scopes required for Databricks access.
  - Note: Remove the offline_access scope as Sigma's documentation states that it is not required for general usage.
- Metadata URL: Enter the OAuth URL from your Databricks SQL warehouse connection details (as seen in the picture above) followed by “.well-known/openid-configuration”.
  - Example Metadata URL: https://dbc-fd9073bc-88f4.cloud.databricks.com/oidc/.well-known/openid-configuration
- Client ID: Provide the client ID from Databricks App Connection collected in Part 1, Step 5.
- Client Secret: Enter the client secret from Databricks App Connection collected in Part 1, Step 5.
Service Account Configuration
- Optionally, set up a service account for cases where workbooks or embedded dashboards must run with service credentials rather than user OAuth.
Write Access
- Optionally, toggle "Write Access" to enable features like materializations or warehouse views. Setting up write access requires dedicated databases, schemas, and permissions in Databricks. See Configure OAuth with write access for prerequisites.
Other Connection Features
- Adjust Connection timeout (default: 120 seconds).
- Enable or disable Use friendly names to control how Sigma formats column names.
- Enable Hive metastore if you want Sigma to see and sync your hive_metastore catalog (Turned off by default).
Click Create to save the Databricks OAuth connection.

This is what a sample Connection looks like within Sigma after setup.

Once saved, users will authenticate through your organization’s identity provider, and credentials will be managed via OAuth for secure access.

Part 3: Test the Connection

After creation, your connection will be available in the Connections list, and you can start building Sigma analyses using Databricks as your data source.

Once you click on your connection, you will be prompted to Sign in with OAuth.

Once you sign in, you will be able to see your Databricks data and start building analyses using Databricks as your data source.

Troubleshooting

For any connectivity errors or issues, please reach out to your designated Sigma or Databricks point of contact for assistance.

Conclusion

Connecting Sigma to Databricks SQL using OAuth provides maximum security, seamless management, and granular governance. This integration empowers your entire organization to make data-driven decisions with confidence, backed by the full power of the Databricks Data Intelligence Platform.