cancel
Showing results for 
Search instead for 
Did you mean: 
Administration & Architecture
Explore discussions on Databricks administration, deployment strategies, and architectural best practices. Connect with administrators and architects to optimize your Databricks environment for performance, scalability, and security.
cancel
Showing results for 
Search instead for 
Did you mean: 

Forum Posts

ziad
by New Contributor III
  • 2061 Views
  • 4 replies
  • 1 kudos

Resolved! Creating Azure Databricks Workspace Without NAT Gateway

Hello,Recently, when I create a new Databricks workspace on Azure, it automatically create a NAT Gateway which incurs additional cost !When creating the workspace, I don't choose secure cluster connectivity, so I'm expecting not to have a NAT Gateway...

nat 1.jpg nat 2.jpg
  • 2061 Views
  • 4 replies
  • 1 kudos
Latest Reply
szymon_dybczak
Esteemed Contributor III
  • 1 kudos

Hi @ziad ,Do you create workspace with secure cluster connectivity? According to the documentation: If you use secure cluster connectivity with the default VNet that Azure Databricks creates, Azure Databricks automatically creates a NAT gateway for o...

  • 1 kudos
3 More Replies
m997al
by Contributor III
  • 3944 Views
  • 1 replies
  • 0 kudos

Cannot manage permissions for Databricks workspace disconnected from metastore

So in the Databricks Account console, under "Workspaces", I am trying to configure permissions for a specific workspace that was formerly attached to a metastore, but the metastore is now deleted (on purpose).No errors show up in the Account console ...

  • 3944 Views
  • 1 replies
  • 0 kudos
itsakhil
by New Contributor
  • 523 Views
  • 2 replies
  • 0 kudos

workspace creation with public IP enabled ( "is_no_public_ip_enabled": false )

Hi All,Am trying to create a workspace on my account which has premium subscription managing through aws market place with is_no_public_ip_enabled": false and am getting below mentioned error.MALFORMED_REQUEST: workspace creation with public IP enabl...

  • 523 Views
  • 2 replies
  • 0 kudos
Latest Reply
Rishabh_Tiwari
Databricks Employee
  • 0 kudos

Hi @itsakhil , Thank you for reaching out to our community! We're here to help you.  To ensure we provide you with the best support, could you please take a moment to review the response and choose the one that best answers your question? Your feedba...

  • 0 kudos
1 More Replies
Pawel_Pilkowski
by New Contributor II
  • 1213 Views
  • 2 replies
  • 0 kudos

Disable Vector Search

Hello all,Is there a way to disable Vector Search? It's not vetted in my organization and I don't want the users to be able to create it.Best,Pawel

  • 1213 Views
  • 2 replies
  • 0 kudos
Latest Reply
Rishabh_Tiwari
Databricks Employee
  • 0 kudos

Hi @Pawel_Pilkowski , Thank you for reaching out to our community! We're here to help you.  To ensure we provide you with the best support, could you please take a moment to review the response and choose the one that best answers your question? Your...

  • 0 kudos
1 More Replies
Phani1
by Valued Contributor II
  • 1766 Views
  • 2 replies
  • 1 kudos

Databricks platform setup best practices in Azure

Could you please help me on the prerequisites, configuring steps and best practices for the data bricks platform and workspace setup in Azure(include  subscription, subnet /network configuration ,storage configuration best practices etc).

  • 1766 Views
  • 2 replies
  • 1 kudos
Latest Reply
Rishabh_Tiwari
Databricks Employee
  • 1 kudos

Hi @Phani1 , Thank you for reaching out to our community! We're here to help you.  To ensure we provide you with the best support, could you please take a moment to review the response and choose the one that best answers your question? Your feedback...

  • 1 kudos
1 More Replies
Mathias
by New Contributor II
  • 8062 Views
  • 6 replies
  • 3 kudos

Different settings per target with Asset bundles

When generating the standard setup with databricks bundle init we will get databricks.yml that references resources/*. The targets are set in the databricks.yml and the resources (pipelines and jobs) are set in different files.I have dlt pipelines th...

  • 8062 Views
  • 6 replies
  • 3 kudos
Latest Reply
MohsenJ
Contributor
  • 3 kudos

I also like to know the solution to this problem

  • 3 kudos
5 More Replies
robmcclain
by New Contributor II
  • 1010 Views
  • 2 replies
  • 1 kudos

Resolved! Using the API, get the list of the schemas and tables a group or user has permissions for

I am attempting to use the Databricks API to get a list of the schemas and tables a group or user has permissions for.  Is this possible?  Is there another method I should be using instead?I see the Unity Catalog > Grants > Get permissions endpoint c...

  • 1010 Views
  • 2 replies
  • 1 kudos
Latest Reply
robmcclain
New Contributor II
  • 1 kudos

Thanks, @Walter_C.  In my case, I was able to get the data I needed by using the Databricks SQL Driver for Node.js , querying the information_schema.table_privileges table.

  • 1 kudos
1 More Replies
lgepp11
by New Contributor III
  • 5062 Views
  • 5 replies
  • 0 kudos

Azure Entra SSO Error: Your user has not been registered

I have set up SSO within databricks and automatic user provisioning with Azure Entra and confirmed it is working for all users. However 1 user is presented with this when signing in. The user is in the enterprise app within Azure Entra and the user i...

lgepp11_0-1696914264539.png
Administration & Architecture
azure
Entra
Error
Sign In
sso
  • 5062 Views
  • 5 replies
  • 0 kudos
Latest Reply
nj28sharp
New Contributor II
  • 0 kudos

Figured out the issue, it seems like Email is case sensitive 

  • 0 kudos
4 More Replies
ShankarM
by Contributor
  • 1142 Views
  • 0 replies
  • 1 kudos

Databricks Clean Room costing

hi,Can you throw some light on how the compute, data sharing costing is done for various scenarios:1. Collaborator 1 and Collaborator 2 are having Databricks accounts in the same region and same cloud. Is there a DBU cost and who will pay for it? I a...

  • 1142 Views
  • 0 replies
  • 1 kudos
a_user12
by New Contributor II
  • 1650 Views
  • 2 replies
  • 0 kudos

Resolved! Databricks Spot Instance: Completion Guarantee

Databricks allows to use spot instances for worker nodes. I consider to use them for interactive clusters. Do I have a gurantee that code will be completed without any errors even if spot instances are evicted? I would accept execution delays but no ...

a_user12_5-1719901164567.png
  • 1650 Views
  • 2 replies
  • 0 kudos
Latest Reply
imsabarinath
New Contributor III
  • 0 kudos

You could explore their "SPOT_WITH_FALLBAK" feature. If you don't want your jobs to fail because of eviction but this currently is not supported with interactive clusters. Hoping that they may extend this to all compute options soonCreate a pipeline ...

  • 0 kudos
1 More Replies
alm
by New Contributor III
  • 4660 Views
  • 4 replies
  • 0 kudos

Show all privileges granted to principal

Given the name of a principal in Databricks (I'm using account-level groups) is there an easy way to query or in other way obtain all privileges granted to this principal?I know I can obtain the information by querying in several of the system.inform...

  • 4660 Views
  • 4 replies
  • 0 kudos
Latest Reply
sakthi_sujitha
Databricks Employee
  • 0 kudos

This link will provide details on how to verify all the privileges granted to Service Principals 

  • 0 kudos
3 More Replies
sk_databricks
by New Contributor
  • 1101 Views
  • 1 replies
  • 0 kudos

Databricks Cache Options

Hi,We are working on Databricks solution hosted on AWS. We are exploring the caching options in Databricks. Apart from the Databricks cache and spark cache? What are the options? Is it feasible to use 3rd party Cache solutions like AWS Elastic Cache ...

  • 1101 Views
  • 1 replies
  • 0 kudos
Latest Reply
Walter_C
Databricks Employee
  • 0 kudos

Databricks provides several caching options to enhance performance by minimizing Input and Output (I/O) read and write operations. These include: Databricks Disk Cache: This cache accelerates data reads by creating copies of remote Parquet data file...

  • 0 kudos
ccsong
by New Contributor II
  • 1912 Views
  • 3 replies
  • 0 kudos

Resolved! How could we share the Databricks ML runtime cluster among users when enable Unity Catalog

Hi team,Currently, we use the Databricks ML runtime to run our workflows and sometimes do the EDA. What we need is that we want to create a Databricks ML runtime for the team to share. When enabling Unity Catalog, how could we create a shared ML runt...

  • 1912 Views
  • 3 replies
  • 0 kudos
Latest Reply
Walter_C
Databricks Employee
  • 0 kudos

Right now there is not plan to support ML runtime in shared clusters. Engineering is working on additional solutions but no ETA is currently available.In regards why it is not supported, principal reason is due to isolation which is not available in ...

  • 0 kudos
2 More Replies
KevinGagnon
by New Contributor
  • 790 Views
  • 0 replies
  • 0 kudos

Delta live table : run_as

Does Databricks have any plans to decouple the owner from the "run_as" identity in Delta Live Table like it can be done in jobs?The problem arise specially when using DABs. The service principal used to deploy DLTs shouldn't be the owner AND the runn...

  • 790 Views
  • 0 replies
  • 0 kudos
macmiller1
by New Contributor II
  • 3439 Views
  • 0 replies
  • 0 kudos

Pass secret in spark config when value is in form a.b.c={{secrets/scope/secret}}

I am configuring the Cluster for a spark-submit task and I am trying to specify `spark.executor.extraJavaOptions a.b.c={{secrets/scope/secret}}` but the literal {{secrets/scope/secret}} is being passed in rather than the secret value itself.I know th...

  • 3439 Views
  • 0 replies
  • 0 kudos

Connect with Databricks Users in Your Area

Join a Regional User Group to connect with local Databricks users. Events will be happening in your city, and you won’t want to miss the chance to attend and share knowledge.

If there isn’t a group near you, start one and help create a community that brings people together.

Request a New Group
Labels