- 2282 Views
- 5 replies
- 3 kudos
Resolved! Connecting Azure databricks with firewall enabled Azure storage account
Hi I am trying to connect from Azure Databrick workspace to Azure gen2 storage account securely. The storage account is set up with these options1. Enabled from selected virtual networks and IP addresses- we whitelisted few ips 2. Added Microsoft.Dat...
- 2282 Views
- 5 replies
- 3 kudos
- 3 kudos
I am having exact issue as @trailblazer , that if I enable traffic for all network, I can read/write to storage account, if I only allow selected network, including the VNet, then it doesn't. I am using Serverless setup. I also followed the firewall ...
- 3 kudos
- 851 Views
- 1 replies
- 1 kudos
Resolved! Job Notifications specifically on Succeeded with Failures
Hi everyone,I have a set of jobs that always execute the last task regardless of whether the previous ones failed or not (using the ‘ALL done’ execution dependency).When moving to production and wanting to enable notifications, there is no option to ...
- 851 Views
- 1 replies
- 1 kudos
- 1 kudos
Databricks does not provide a direct way to distinguish or send notifications specifically for a "Succeeded with failures" state at the job level—the job is classified as "Success" even when some upstream tasks have failed, if the last (leaf) task is...
- 1 kudos
- 685 Views
- 2 replies
- 3 kudos
Resolved! Error when trying to destory databricks_permissions with OpenTofu
Hi,In our company's project we created a databricks_user for a service account (which is needed for our deployment process) via OpenTofu and afterwards adjusted permissions to that "user's" user folder using the databricks_permissions resource.resour...
- 685 Views
- 2 replies
- 3 kudos
- 3 kudos
Hi @MiriamHundemer , The issue occurs because the owner of the home folder (in this case, the databricks_user.databricks_deployment_sa service account) often has an unremovable CAN_MANAGE permission on its own home directory. When OpenTofu attempts t...
- 3 kudos
- 614 Views
- 4 replies
- 1 kudos
Resolved! Deply databricks workspace on azure with terraform - failed state: legacy access
I'm trying to deploy a workspace on azure via terraform and i'm getting the following error:"INVALID_PARAMETER_VALUE: Given value cannot be set for workspace~<id>~default_namespace_ws~ because: cannot set default namespace to hive_metastore since leg...
- 614 Views
- 4 replies
- 1 kudos
- 1 kudos
I found the issue, The setting automatically assigned workspaces to this metastore was checked. Unchecking this and manually assigning the metastore worked.
- 1 kudos
- 350 Views
- 1 replies
- 1 kudos
Clarification on Unity Catalog Metastore - Metadata and storage
Where does the Unity Catalog metastore metadata actually reside?Is it stored and managed in the Databricks account (control plane)?Or does it get stored in the customer-managed S3 bucket when we create a bucket for Unity Catalog metastore?I want to c...
- 350 Views
- 1 replies
- 1 kudos
- 1 kudos
@APJESK Replied here https://community.databricks.com/t5/data-governance/clarification-on-unity-catalog-metastore-metadata-and-storage/td-p/133389
- 1 kudos
- 540 Views
- 3 replies
- 3 kudos
Resolved! Databricks GCP login with company account
I created gmail on my company account, then I logged in in gcp databricks, till yesterday it was working fine, yesterday I logged in into gmail account, there it asked for another gmail id, so I have provided new, but today I am not able to login usi...
- 540 Views
- 3 replies
- 3 kudos
- 3 kudos
Hello @xavier_db! Were you able to get this login issue resolved? If yes, it would be great if you could share what worked for you so others facing the same problem can benefit as well.
- 3 kudos
- 573 Views
- 4 replies
- 6 kudos
Resolved! Is there a way to register S3 compatible tables?
Hi everyone,I have successfully registered AWS S3 tables in unity catalog, but I would like to register S3-compatible as well.But, to create an EXTERNAL LOCATION in unity catalog, it seems I must register a credential. But the only suported credentia...
- 573 Views
- 4 replies
- 6 kudos
- 6 kudos
Hey @tabasco, did you check out the External Table documentation in the Databricks AWS docs?External Locationhttps://docs.databricks.com/aws/en/sql/language-manual/sql-ref-external-locationsCredentialshttps://docs.databricks.com/aws/en/sql/language-m...
- 6 kudos
- 297 Views
- 1 replies
- 1 kudos
Workload identity federation policy
Dear allCan I create a single workload federation policy for all devops pipelines?Our set-up : we have code version controlled in Github repos. And, we use Azure DevOps pipelines to authenticate with Databricks via a service principal currently and d...
- 297 Views
- 1 replies
- 1 kudos
- 1 kudos
Hi @noorbasha534 ,In docs they are giving following example of subject requirements for Azure Devops. So, the subject (sub) claim must uniquely identify the workload. So as long as all of your pipelines resides in the same organization, same project ...
- 1 kudos
- 2247 Views
- 6 replies
- 9 kudos
Resolved! Is it possible restore a deleted catalog and schema
Is it possible restore a deleted catalog and schema.if CASCADE is used even though schemas and tables are present in catalog, catalog will be dropped.Is it possible to restore catalog or is possible to restrict the use of CACADE command.Thank you.
- 2247 Views
- 6 replies
- 9 kudos
- 9 kudos
@Louis_Frolio I cannot click any "Accept as Solution" button, as I was not the one creating the post, I believe
- 9 kudos
- 462 Views
- 1 replies
- 1 kudos
Resolved! Dev/Prod Environments in AWS: Separate Accounts vs. Separate Workspaces?
Hello everyone,I'm looking for some advice on best practices. When setting up development and production environments for Databricks on AWS, is it better to use completely separate AWS accounts, or is it sufficient to use separate workspaces within a...
- 462 Views
- 1 replies
- 1 kudos
- 1 kudos
Hi @tana_sakakimiya , I allow myself copy and paste brilliant answer on similar question provided by user Isi:"Option A: Multiple Databricks accounts and multiple AWS accountsThis model offers the highest level of isolation. Each environment lives in...
- 1 kudos
- 4521 Views
- 3 replies
- 4 kudos
Resolved! Enable the "Billable Usage Download' Accounts API on our Azure
Hi Databricks Support,Could you please confirm whether the Billable Usage Download endpoint can be enabled on Azure Databricks account, similar to how it’s available on AWS/GCP? If yes, what are the steps that should be followed for the same.. else ...
- 4521 Views
- 3 replies
- 4 kudos
- 3456 Views
- 1 replies
- 1 kudos
Mismatch cuda/cudnn version on Databricks Runtime GPU ML version
I have a cluster on Databricks with configuration Databricks Runtime Version16.4 LTS ML Beta (includes Apache Spark 3.5.2, GPU, Scala 2.12), and another cluster with configuration 16.0 ML (includes Apache Spark 3.5.2, GPU, Scala 2.12). According to...
- 3456 Views
- 1 replies
- 1 kudos
- 1 kudos
There could be library related conflicts in 16.0ML that got fixed in 16.4ML. I would always recommend to use the LTS version. Thanks
- 1 kudos
- 342 Views
- 2 replies
- 1 kudos
DataBricks JDBC driver fails when unsupported property is submitted
Hello!The Databricks jdbc driver applies a property that is not supported by the connector as a Spark server-side property for the client session. How can I avoid that ? With some tools I do not have 100% control, eg they may add a custom jdbc connec...
- 342 Views
- 2 replies
- 1 kudos
- 1 kudos
Hi @Kaitsu , The documentation mentions, If you specify a property that is not supported by the connector, then the connector attempts to apply the property as a Spark server-side property for the client session. Unlike many other JDBC drivers that ...
- 1 kudos
- 3931 Views
- 2 replies
- 2 kudos
Pre-loading docker images to cluster pool instances still requires docker URL at cluster creation
I am trying to pre-load a docker image to a Databricks cluster pool instance.As per this article I used the REST API to create the cluster pool and defined a custom Azure container registry as the source for the docker images.https://learn.microsoft....
- 3931 Views
- 2 replies
- 2 kudos
- 2 kudos
@NadithK Pre-loading Docker images to cluster pool instances is for performance optimization (faster cluster startup), but you still must specify the Docker image in your cluster configuration. The pre-loading doesn't eliminate the requirement to dec...
- 2 kudos
- 1120 Views
- 4 replies
- 3 kudos
Resolved! Databricks Default package repositories
I have added an extra-index-url in the default package repository in databricks which points to a repository in azure artifact. The libraries from it are getting installed on job cluster but is not working on the all purpose cluster. Below is the rel...
- 1120 Views
- 4 replies
- 3 kudos
- 3 kudos
Hello @tariq! Did the suggestions shared above help address your issue? If so, please consider marking one of the responses as the accepted solution. If you found a different approach that worked for you, sharing it with the community would be really...
- 3 kudos
