Databricks

knawara · ‎02-20-2023

Hello,

What is the relationship between Databricks Account (as described in [1]) and Azure resources? Is Databricks Account created per Azure account? Or per Azure tenant? Or maybe per subscription?

[1] https://learn.microsoft.com/en-us/azure/databricks/administration-guide/account-settings/

Kaniz · ‎02-22-2023

Hi @Chris Nawara, Databricks is a cloud-based data analytics platform that provides a collaborative environment for data engineering, machine learning, and analytics. Databricks can be deployed on several cloud platforms, including Microsoft Azure.

To use Databricks on Azure, you need an Azure subscription and a Databricks account. Your Databricks account is linked to your Azure subscription, and you can create and manage Databricks workspaces in the Azure portal.

When you create a Databricks workspace in Azure, it is designed as an Azure resource. The Databricks workspace provides a cluster of computing resources for processing data and running analytics jobs.

The workspace is also linked to other Azure resources, such as storage accounts and virtual networks, which store data and provide connectivity to other Azure services.

You can manage the resources associated with your Databricks workspaces, such as clusters and notebooks, using the Azure portal or the Databricks UI. You can also use Azure Resource Manager templates to programmatically deploy and manage Databricks workspaces and associated resources.

In summary, a Databricks account is required to use Databricks on Azure. The Databricks workspace is created as an Azure resource linked to other Azure resources for data storage and connectivity.

knawara · ‎02-22-2023

Hi @Kaniz Fatma , thank you for your response.

As far as I understand the hierarchy in Azure is as follows:

Azure Account -> Tenancy -> Subscription -> Resource Group -> Resource

In my account I have multiple tenants and they have multiple subscription. You said that "Your Databricks account is linked to your Azure subscription". Does it mean that each subscription needs to have a dedicated Databricks account?

Kaniz · ‎02-22-2023

Hi @Chris Nawara , Each Azure subscription does not need a dedicated Databricks account.

When you sign up for a Databricks account on Azure, you create a single account linked to your Azure subscription. This account is used to manage your Databricks workspaces and resources, and you can create multiple workspaces within the same account.

Each workspace is a different environment with its resources, such as clusters, notebooks, and libraries. You can create multiple workspaces within the same Databricks account and share resources between workspaces if needed.

So, you can have multiple workspaces in a single Databricks account associated with different Azure subscriptions or projects. This allows you to centralize your Databricks usage and manage your resources more efficiently.

knawara · ‎02-22-2023

Hi @Kaniz Fatma !

Is account created implicitly? Or is it created explicitly and then linked to workspaces?

To give you some background for the question - we're part of a big organisation, with a lot of different teams. Each team gets (from a central Azure team) their own Azure subscription with a full admin access. Azure root account and tenant is shared by entire organization.

Now we'd like to enable Unity Catalog for our workspaces, but for that account-level access is required. If Databricks account can be scoped down to an Azure subscription, we're good - the central Azure team could give us admin rights to the account. But if account is, by definition, linked to workspaces across subscriptions and tenants they can't - and it's a big deal to get it set up in a secure way. This most likely means no Unity Catalog for us

Kaniz · ‎02-24-2023

Hi @Chris Nawara, In Databricks, an account is created explicitly when you sign up for a Databricks subscription. Once an account is created, you can create and manage workspaces within that account.

If your organization cannot grant you account-level access, then you may not be able to use Unity Catalog with your Databricks workspaces.

Anonymous · ‎04-21-2023

Hi @Chris Nawara

Hope everything is going great.

Just wanted to check in if you were able to resolve your issue. If yes, would you be happy to mark an answer as best so that other members can find the solution more quickly? If not, please tell us so we can help you.

Cheers!