cancel
Showing results for 
Search instead for 
Did you mean: 
Administration & Architecture
Explore discussions on Databricks administration, deployment strategies, and architectural best practices. Connect with administrators and architects to optimize your Databricks environment for performance, scalability, and security.
cancel
Showing results for 
Search instead for 
Did you mean: 

Forum Posts

chethankumar
by New Contributor III
  • 1177 Views
  • 3 replies
  • 1 kudos

Resolved! How to add existing recipient to existing delta share

I am facing an issue while adding a recipient to Delta Share using Terraform. The owner of my recipient is a group, not an individual user. I'm running this Terraform script using a service principal member of that group. However, I'm encountering th...

  • 1177 Views
  • 3 replies
  • 1 kudos
Latest Reply
chethankumar
New Contributor III
  • 1 kudos

I was able to fix the issue. The problem was that the service principal I was using didn’t have the correct permissions assigned

  • 1 kudos
2 More Replies
eirik_morken
by New Contributor III
  • 2666 Views
  • 10 replies
  • 0 kudos

Resolved! Python function "go to definition" and "peek definition" do not work

When using notebooks with python in Databricks I would really like to easily see the defintion of the functions I am using within the editor. Which the "Go to definition (F12)" and "Peek definition" options when right clicking on the functions will h...

  • 2666 Views
  • 10 replies
  • 0 kudos
Latest Reply
eirik_morken
New Contributor III
  • 0 kudos

This has now been resolved and is working as expected. Do not know why or how, but something has changed that made it work

  • 0 kudos
9 More Replies
JameDavi_51481
by Contributor
  • 3719 Views
  • 5 replies
  • 2 kudos

Adhoc workflows - managing resource usage on shared clusters

We run a shared cluster that is used for general purpose adhoc analytics, which I assume is a relatively common use case to try to keep costs down. However, the technical experience of users of this cluster varies a lot, so we run into situations whe...

  • 3719 Views
  • 5 replies
  • 2 kudos
Latest Reply
sshssh
New Contributor II
  • 2 kudos

Hi, @JameDavi_51481 , were you able to figure something out?Planning a Databricks migration and realized we might need something similar too. 

  • 2 kudos
4 More Replies
ibanezcp88
by New Contributor II
  • 614 Views
  • 2 replies
  • 0 kudos

Sporadic HTTP failure with SQL Serverless (bug?)

Our SQL Serverless installation has sporadic failures to our blob container in Azure.  The blob container is locked down to a vnet, and we are using the private endpoint to enable serverless access.  It will work fine for several hours, and then show...

  • 614 Views
  • 2 replies
  • 0 kudos
Latest Reply
ibanezcp88
New Contributor II
  • 0 kudos

I've confirmed all of that.  This seems like an AI generated response.  It seems more likely that Databricks rolled out a feature a week ago that is causing instability in the serverless warehouses.  Any other specific things to check would be apprec...

  • 0 kudos
1 More Replies
Rameshkumar85
by New Contributor
  • 960 Views
  • 1 replies
  • 0 kudos

I want to create custom tag in cluster policy so that clusters created using that policy get those

"I want to create custom tags in a cluster policy so that clusters created using this policy will automatically include those tags for billing purposes. Consider the following example:"cluster_type": {"type": "fixed","value": "all-purpose"},"custom_t...

  • 960 Views
  • 1 replies
  • 0 kudos
Latest Reply
Walter_C
Databricks Employee
  • 0 kudos

Are you having any issue while running this code in the policy?

  • 0 kudos
Phani1
by Valued Contributor II
  • 1027 Views
  • 1 replies
  • 0 kudos

Databricks Serverless best practices

Hi All, We are configuring a Databricks serverless that adjusts according to the workload type,like choosing different cluster sizes such as extra small ,small ,large etc, and auto scale option.We're also looking at the average time it takes to compl...

  • 1027 Views
  • 1 replies
  • 0 kudos
Latest Reply
Walter_C
Databricks Employee
  • 0 kudos

You can refer to our Serverless Compute Best practices: https://docs.databricks.com/en/compute/serverless/best-practices.htmlIf you refer to the Serverless Warehouses you can refer to https://docs.databricks.com/en/compute/sql-warehouse/warehouse-beh...

  • 0 kudos
SALAHUDDINKHAN
by New Contributor II
  • 609 Views
  • 1 replies
  • 0 kudos

Manage Account option

HiI have created a premium databricks workspace on my azure free trial account and also have the global administrator role on my azure acount. I have setup all the necessary configurations like by providing the role of storage data blob contributor t...

  • 609 Views
  • 1 replies
  • 0 kudos
Latest Reply
Alberto_Umana
Databricks Employee
  • 0 kudos

Hello @SALAHUDDINKHAN, If you are unable to see the "Manage Account" option, it is likely that you do not have the necessary account admin privileges. Please ensure you have the required permissions indicated here: https://learn.microsoft.com/en-us/a...

  • 0 kudos
harvey-c
by New Contributor III
  • 11770 Views
  • 2 replies
  • 1 kudos

Resolved! Any way to move the unity catalog to a new external storage location?

Dear Databricks CommunityThe question is about changing an existing unity catalog to a new storage location. For example: With an existing unity catalog (i.e. catalog1)  includeing schemas and volumes. The catalog is based on an external location (i....

  • 11770 Views
  • 2 replies
  • 1 kudos
Latest Reply
NandiniN
Databricks Employee
  • 1 kudos

https://docs.databricks.com/ja/sql/language-manual/sql-ref-syntax-ddl-alter-location.html

  • 1 kudos
1 More Replies
ashish_modi
by New Contributor II
  • 1189 Views
  • 3 replies
  • 1 kudos

cloud_infra_costs

I was looking at the system catalog and realized that there is an empty table called cloud_infra_costs. Could you tell me what is this for and why it is empty?  

ashish_modi_0-1730131553537.png
  • 1189 Views
  • 3 replies
  • 1 kudos
Latest Reply
ashish_modi
New Contributor II
  • 1 kudos

Thanks for replying. This makes sense. Any idea why it is empty and what to do to populate this?

  • 1 kudos
2 More Replies
Fkebbati
by New Contributor II
  • 2953 Views
  • 5 replies
  • 2 kudos

Ingress/Egress private endpoint

Hello ,We have configured our Databricks environment with private endpoint connections injected into our VNET, which includes two subnets (public and private). We have disabled public IPs and are using Network Security Groups (NSGs) on the subnet, as...

  • 2953 Views
  • 5 replies
  • 2 kudos
Latest Reply
JakubSkibicki
Contributor
  • 2 kudos

@Fkebbati First, traffic cost in Azure are not reported as a separate Resource Type, but appended to main resource causing the traffic. If you want to distinguish them use for instance Service Name. In this case traffic cost is appended to Databricks...

  • 2 kudos
4 More Replies
Sadam97
by New Contributor III
  • 1459 Views
  • 3 replies
  • 0 kudos

Databricks (GCP) Cluster not resolving Hostname into IP address

we have #mongodb hosts that must be resolved to private internal loadbalancer ips ( of another cluster ), and that we are unable to add host aliases in the Databricks GKE cluster in order for the spark to be able to connect to a mongodb and resolve t...

  • 1459 Views
  • 3 replies
  • 0 kudos
Latest Reply
NandiniN
Databricks Employee
  • 0 kudos

Also found this - https://community.databricks.com/t5/data-engineering/why-i-m-getting-connection-timeout-when-connecting-to-mongodb/m-p/14868

  • 0 kudos
2 More Replies
ccsong
by New Contributor II
  • 2426 Views
  • 3 replies
  • 0 kudos

Data leakage risk happened when we use the Azure Databricks workspace

Context:We are utilizing an Azure Databricks workspace for data management and model serving within our project, with delegated VNet and subnets configured specifically for this workspace. However, we are consistently observing malicious flow entries...

Screenshot 2024-10-16 at 18.14.45.png Screenshot 2024-10-16 at 18.15.19.png Screenshot 2024-10-16 at 18.18.24.png
  • 2426 Views
  • 3 replies
  • 0 kudos
Latest Reply
Alberto_Umana
Databricks Employee
  • 0 kudos

Hello everyone! We have worked with our security team, Microsoft, and other customers who have seen similar log messages. This log message is very misleading, as it appears to state that the malicious URI was detected within your network â€” this would...

  • 0 kudos
2 More Replies
peachum
by New Contributor
  • 789 Views
  • 1 replies
  • 0 kudos

Unable to add a databricks permission to existing policy

Hi,   We're using databricks provider v1.49.1 to manager our Azure databricks cluster and other resources.    Having an issue setting permissions with the databricks terraform resource "databricks_permissions" where the error indicates that the clust...

  • 789 Views
  • 1 replies
  • 0 kudos
Latest Reply
Walter_C
Databricks Employee
  • 0 kudos

Is this cluster policy a custom policy? If you try for testing purposes to modify it in the UI does it allows you to  

  • 0 kudos
Ali_Ahmad
by New Contributor II
  • 2134 Views
  • 1 replies
  • 0 kudos

Compute terminated. Reason: Control Plane Request Failure

Hi,I have started to get this error: Failed to get instance bootstrap steps from the Databricks Control Plane. Please check that instances have connectivity to the Databricks Control Plane. and I am suspecting it has to do with networking.I am just a...

  • 2134 Views
  • 1 replies
  • 0 kudos
Latest Reply
Walter_C
Databricks Employee
  • 0 kudos

Do you still facing issues? I would suggest to check the Securitygroups and make sure they match with:https://docs.databricks.com/en/security/network/classic/customer-managed-vpc.html#security-groupsAdditionally check if the inbound and outbound addr...

  • 0 kudos
NielsMH
by New Contributor III
  • 1902 Views
  • 1 replies
  • 0 kudos

Updating databricks git repo from github action - how to

HiMy company is migrating from azuredevops to github and we have a pipeline in azuredevops which updates/syncs databricks repos whenever a pull request is made to the development branch. The azure devops pipeline (which works) looks like this: trigge...

  • 1902 Views
  • 1 replies
  • 0 kudos
Latest Reply
Walter_C
Databricks Employee
  • 0 kudos

It seems like the issue you're encountering is related to missing Git provider credentials when trying to update the Databricks repo via GitHub Actions. Based on the context provided, here are a few steps you can take to resolve this issue:   Verify...

  • 0 kudos