- 2374 Views
- 3 replies
- 1 kudos
Terraform - Azure Databricks workspace without NAT gateway
Hi all,I have experienced an increase in costs - even when not using Databricks compute.It is due to the NAT-gateway, that are (suddenly) automatically deployed.When creating Azure Databricks workspaces using Terraform:A NAT-gateway is created. When ...
- 2374 Views
- 3 replies
- 1 kudos
- 1 kudos
In Azure Databricks, a NAT Gateway will be required (by Microsoft) for all egress from VMs, which affects Databricks compute: Azure updates | Microsoft Azure
- 1 kudos
- 3102 Views
- 2 replies
- 0 kudos
Resolved! External Locations to Azure Storage via Private Endpoint
When working with Azure Databricks (with VNET injection) to connect securely to an Azure Storage account via private endpoint, there's a few locations it needs to connect to, firstly the vnet that databricks is connected to, which works well when con...
- 3102 Views
- 2 replies
- 0 kudos
- 0 kudos
I've read that in the documentation, and when I now tried with an Access Connector for Azure Databricks instead of my own service principal, it seems to have worked, shockingly, even if I completely block network access on the storage account with ze...
- 0 kudos
- 1677 Views
- 5 replies
- 0 kudos
Any documentation mentioning connectivity from Azure SQL database connectivity to Azure Databricks
Any documentation available to connect from the Azure SQL database to Azure Databricks SQL workspace. We created a SQL warehouse personal access token for a user in a different team who can connect from his on-prem SQL DB to Databricks using the conn...
- 1677 Views
- 5 replies
- 0 kudos
- 0 kudos
Here are some considerations: SQL Trigger: Define a trigger in Azure SQL that activates on specific DML operations (e.g., INSERT, UPDATE). External Call: The trigger can log events to an intermediate service (like a control table or Event Grid). ...
- 0 kudos
- 1398 Views
- 3 replies
- 2 kudos
Resolved! Compute page Resolver error
I have an issue with my Databricks workspaceWhen i try to attach a notebook to my cluster, the cluster seems to have disappeared.After navigating to the Compute page, it gives me a 'Resolver Error' and no further information. None of the computes/clu...
- 1398 Views
- 3 replies
- 2 kudos
- 2 kudos
Hello @Maxadbuser! This was a known issue, and the engineering team has implemented a fix. Please check and confirm if you're still experiencing the problem.
- 2 kudos
- 1264 Views
- 3 replies
- 1 kudos
Resolved! NAT Gateway IP update
Hi, My Databricks (Premium) account was deployed on AWS.It was provisioned a few months ago from the AWS MarketPlace with the QuickStart method, based on CloudFormation. The NAT Gateway initially created by the CloudFormation stack has been incidenta...
- 1264 Views
- 3 replies
- 1 kudos
- 1 kudos
Thanks for your answer.Yes I did all of that, as well as allowing the traffic coming from my NAT Instance as an inbound rule of the Security Group of the private instances (and the other way around), and no restriction on the outbound traffic.I suspe...
- 1 kudos
- 30246 Views
- 6 replies
- 1 kudos
Resolved! Delete Databricks account
Hi everyone, as in the topic, I would like to delete unnnecesarily created account. I have found outdated solutions (e.g. https://community.databricks.com/t5/data-engineering/how-to-delete-databricks-account/m-p/6323#M2501), but they do not work anym...
- 30246 Views
- 6 replies
- 1 kudos
- 1 kudos
Hi @Edyta ,FOR AWS:Manage your subscription | Databricks on AWS" Before you delete a Databricks account, you must first cancel your Databricks subscription and delete all Unity Catalog metastores in the account. After you delete all metastores associ...
- 1 kudos
- 726 Views
- 1 replies
- 0 kudos
Is it possible to let multiple Apps share the same compute?
Apparently, for every app deployed on Databricks, a separate VM is allocated, costing 0.5 DBU/hour. This seems inefficient, why can't a single VM support multiple apps? It feels like a waste of money and resources to allocate independent VMs per app ...
- 726 Views
- 1 replies
- 0 kudos
- 0 kudos
Hi @Aminsn , You're understanding is correct in that only 1 running app can be deployed per app instance. If you want to maximize the utilization of the compute, one option could be to create a multi-page app, where the landing page will direct users...
- 0 kudos
- 6637 Views
- 2 replies
- 0 kudos
Databricks App : Limitations
I have some questions regarding Databricks App.1) Can we use Framework other than mentioned in documentation( Streamlit,Flask,Dash,Gradio,Shiny).2) Can we allocate compute more than 2 vCPU and 6GB memory to any App.3) Any other programming language o...
- 6637 Views
- 2 replies
- 0 kudos
- 0 kudos
1.) You can use most Python-based application frameworks, including some beyond those mentioned above.(Reference here) 2.) Currently, app capacity is limited to 2 vCPUs and 6 GB of RAM. However, future updates may introduce options for scaling out an...
- 0 kudos
- 1941 Views
- 4 replies
- 2 kudos
Resolved! OAuth API for service user
Is there a way to programmatically create an OAuth secret for for a workspace service principal via API/SDK? As of now, the only way I can see doing this is through UI
- 1941 Views
- 4 replies
- 2 kudos
- 2 kudos
@andreapeterson isn't it the API you are looking for?https://docs.databricks.com/api/azure/account/serviceprincipalsecrets/createIt is an account-level API, but, counterintuitively, when we create service principals in the workspace, they propagate i...
- 2 kudos
- 826 Views
- 1 replies
- 0 kudos
Load assignment during Distributed training
Hi,I wanted to confirm, in a distributed training, if there is any way that I can control what kind/amount of load/data can be send to specific worker nodes, manually ..Or is it completely automatically handled by spark's scheduler, and we don't have...
- 826 Views
- 1 replies
- 0 kudos
- 0 kudos
From what I know, Spark automatically handles how data and workload are distributed across worker nodes during distributed training, you can't manually control exactly what or how much data goes to a specific node. You can still influence the distrib...
- 0 kudos
- 877 Views
- 1 replies
- 0 kudos
Running Driver Intensive workloads on all purpose compute
Recently observed when we run a driver intensive code on a all purpose compute. The parallel runs of the same pattern/kind jobs are getting failedExample:Job triggerd on all purpose compute with compute stats of 4 core and 8 gigs ram for driverLets s...
- 877 Views
- 1 replies
- 0 kudos
- 0 kudos
this will help you # Cluster config adjustmentsspark.conf.set("spark.driver.memory", "16g") # Double current allocationspark.conf.set("spark.driver.maxResultSize", "8g") # Prevent large collects
- 0 kudos
- 1398 Views
- 3 replies
- 3 kudos
Resolved! how to modify workspace creator
Hi. Our 3 workspaces were created by a consultant who is no longer with us. The workspace shows his name still . How can we change it? Also what kind of account should be a creator of a workspace: Service principal, AD account, AD group? Pease see b...
- 1398 Views
- 3 replies
- 3 kudos
- 3 kudos
You can file an Azure ticket and request them to contact Databricks Support, who will contact Databricks Engineering, to change that 'workspace owner'. If you have a Databricks Account Team, they can also file an internal ticket to Databricks Enginee...
- 3 kudos
- 1447 Views
- 4 replies
- 0 kudos
Terraforming Git credentials for service principals
I am terraforming service principals in my Databricks workspace and it works great until I need to assign Git credentials to my SP. In the UI we have these options to configure credentials on service principal page:However the Terraform resource I fo...
- 1447 Views
- 4 replies
- 0 kudos
- 0 kudos
You're a little bit ahead of me in this process, so I haven't tried the solution yet, but it looks like you create a git credential resource for the service principal. This requires a token, which I think must be generated in the console. My refere...
- 0 kudos
- 1141 Views
- 1 replies
- 0 kudos
Unity Catalog: 403 Error When Connecting S3 via IAM Role and Storage Credential
Hi,We're currently setting up Databricks Unity Catalog on AWS. We created an S3 bucket and assigned an IAM role (databricks-storage-role) to give Databricks access.Note: Databricks doesn't use the IAM role directly. Instead, it requires a Storage Cre...
- 1141 Views
- 1 replies
- 0 kudos
- 0 kudos
Have you follow any specific guide for the creation of the same? Are you setting up a Unity Catalog Metastore or the default storage for the workspace?For the Metastore creation have you follow steps in https://docs.databricks.com/aws/en/data-governa...
- 0 kudos
- 3131 Views
- 7 replies
- 1 kudos
How to install (mssql) drivers to jobcompute?
Hello, I'm having this issue with job-computes:The snippet of the code is as follows: 84 if self.conf["persist_to_sql"]: 85 # persist to sql 86 df_parsed.write.format( 87 "com.microsoft.sqlserver.jdbc.spark" 88...
- 3131 Views
- 7 replies
- 1 kudos
- 1 kudos
For a job compute, you would have to go init script route. Can you please highlight, the cause of the failure of library installation via init script?
- 1 kudos
