- 885 Views
- 1 replies
- 0 kudos
Resolved! How to create a function using the functions API in databricks?
https://docs.databricks.com/api/workspace/functions/createThis documentation gives the sample request payload, and one of the fields is type_json, and there is very little explanation of what is expected in this field. What am I supposed to pass here...
- 885 Views
- 1 replies
- 0 kudos
- 0 kudos
Hi @chinmay0924 ,The type_json field describes your function’s input parameters and return type using a specific JSON format. You’ll need to include each parameter’s name, type (like "STRING", "INT", "ARRAY", or "STRUCT"), and position, along with th...
- 0 kudos
- 2323 Views
- 3 replies
- 1 kudos
Terraform - Azure Databricks workspace without NAT gateway
Hi all,I have experienced an increase in costs - even when not using Databricks compute.It is due to the NAT-gateway, that are (suddenly) automatically deployed.When creating Azure Databricks workspaces using Terraform:A NAT-gateway is created. When ...
- 2323 Views
- 3 replies
- 1 kudos
- 1 kudos
In Azure Databricks, a NAT Gateway will be required (by Microsoft) for all egress from VMs, which affects Databricks compute: Azure updates | Microsoft Azure
- 1 kudos
- 2966 Views
- 2 replies
- 0 kudos
Resolved! External Locations to Azure Storage via Private Endpoint
When working with Azure Databricks (with VNET injection) to connect securely to an Azure Storage account via private endpoint, there's a few locations it needs to connect to, firstly the vnet that databricks is connected to, which works well when con...
- 2966 Views
- 2 replies
- 0 kudos
- 0 kudos
I've read that in the documentation, and when I now tried with an Access Connector for Azure Databricks instead of my own service principal, it seems to have worked, shockingly, even if I completely block network access on the storage account with ze...
- 0 kudos
- 1643 Views
- 5 replies
- 0 kudos
Any documentation mentioning connectivity from Azure SQL database connectivity to Azure Databricks
Any documentation available to connect from the Azure SQL database to Azure Databricks SQL workspace. We created a SQL warehouse personal access token for a user in a different team who can connect from his on-prem SQL DB to Databricks using the conn...
- 1643 Views
- 5 replies
- 0 kudos
- 0 kudos
Here are some considerations: SQL Trigger: Define a trigger in Azure SQL that activates on specific DML operations (e.g., INSERT, UPDATE). External Call: The trigger can log events to an intermediate service (like a control table or Event Grid). ...
- 0 kudos
- 1369 Views
- 3 replies
- 2 kudos
Resolved! Compute page Resolver error
I have an issue with my Databricks workspaceWhen i try to attach a notebook to my cluster, the cluster seems to have disappeared.After navigating to the Compute page, it gives me a 'Resolver Error' and no further information. None of the computes/clu...
- 1369 Views
- 3 replies
- 2 kudos
- 2 kudos
Hello @Maxadbuser! This was a known issue, and the engineering team has implemented a fix. Please check and confirm if you're still experiencing the problem.
- 2 kudos
- 1236 Views
- 3 replies
- 1 kudos
Resolved! NAT Gateway IP update
Hi, My Databricks (Premium) account was deployed on AWS.It was provisioned a few months ago from the AWS MarketPlace with the QuickStart method, based on CloudFormation. The NAT Gateway initially created by the CloudFormation stack has been incidenta...
- 1236 Views
- 3 replies
- 1 kudos
- 1 kudos
Thanks for your answer.Yes I did all of that, as well as allowing the traffic coming from my NAT Instance as an inbound rule of the Security Group of the private instances (and the other way around), and no restriction on the outbound traffic.I suspe...
- 1 kudos
- 30010 Views
- 6 replies
- 1 kudos
Resolved! Delete Databricks account
Hi everyone, as in the topic, I would like to delete unnnecesarily created account. I have found outdated solutions (e.g. https://community.databricks.com/t5/data-engineering/how-to-delete-databricks-account/m-p/6323#M2501), but they do not work anym...
- 30010 Views
- 6 replies
- 1 kudos
- 1 kudos
Hi @Edyta ,FOR AWS:Manage your subscription | Databricks on AWS" Before you delete a Databricks account, you must first cancel your Databricks subscription and delete all Unity Catalog metastores in the account. After you delete all metastores associ...
- 1 kudos
- 691 Views
- 1 replies
- 0 kudos
Is it possible to let multiple Apps share the same compute?
Apparently, for every app deployed on Databricks, a separate VM is allocated, costing 0.5 DBU/hour. This seems inefficient, why can't a single VM support multiple apps? It feels like a waste of money and resources to allocate independent VMs per app ...
- 691 Views
- 1 replies
- 0 kudos
- 0 kudos
Hi @Aminsn , You're understanding is correct in that only 1 running app can be deployed per app instance. If you want to maximize the utilization of the compute, one option could be to create a multi-page app, where the landing page will direct users...
- 0 kudos
- 6476 Views
- 2 replies
- 0 kudos
Databricks App : Limitations
I have some questions regarding Databricks App.1) Can we use Framework other than mentioned in documentation( Streamlit,Flask,Dash,Gradio,Shiny).2) Can we allocate compute more than 2 vCPU and 6GB memory to any App.3) Any other programming language o...
- 6476 Views
- 2 replies
- 0 kudos
- 0 kudos
1.) You can use most Python-based application frameworks, including some beyond those mentioned above.(Reference here) 2.) Currently, app capacity is limited to 2 vCPUs and 6 GB of RAM. However, future updates may introduce options for scaling out an...
- 0 kudos
- 1910 Views
- 4 replies
- 2 kudos
Resolved! OAuth API for service user
Is there a way to programmatically create an OAuth secret for for a workspace service principal via API/SDK? As of now, the only way I can see doing this is through UI
- 1910 Views
- 4 replies
- 2 kudos
- 2 kudos
@andreapeterson isn't it the API you are looking for?https://docs.databricks.com/api/azure/account/serviceprincipalsecrets/createIt is an account-level API, but, counterintuitively, when we create service principals in the workspace, they propagate i...
- 2 kudos
- 804 Views
- 1 replies
- 0 kudos
Load assignment during Distributed training
Hi,I wanted to confirm, in a distributed training, if there is any way that I can control what kind/amount of load/data can be send to specific worker nodes, manually ..Or is it completely automatically handled by spark's scheduler, and we don't have...
- 804 Views
- 1 replies
- 0 kudos
- 0 kudos
From what I know, Spark automatically handles how data and workload are distributed across worker nodes during distributed training, you can't manually control exactly what or how much data goes to a specific node. You can still influence the distrib...
- 0 kudos
- 867 Views
- 1 replies
- 0 kudos
Running Driver Intensive workloads on all purpose compute
Recently observed when we run a driver intensive code on a all purpose compute. The parallel runs of the same pattern/kind jobs are getting failedExample:Job triggerd on all purpose compute with compute stats of 4 core and 8 gigs ram for driverLets s...
- 867 Views
- 1 replies
- 0 kudos
- 0 kudos
this will help you # Cluster config adjustmentsspark.conf.set("spark.driver.memory", "16g") # Double current allocationspark.conf.set("spark.driver.maxResultSize", "8g") # Prevent large collects
- 0 kudos
- 1347 Views
- 3 replies
- 3 kudos
Resolved! how to modify workspace creator
Hi. Our 3 workspaces were created by a consultant who is no longer with us. The workspace shows his name still . How can we change it? Also what kind of account should be a creator of a workspace: Service principal, AD account, AD group? Pease see b...
- 1347 Views
- 3 replies
- 3 kudos
- 3 kudos
You can file an Azure ticket and request them to contact Databricks Support, who will contact Databricks Engineering, to change that 'workspace owner'. If you have a Databricks Account Team, they can also file an internal ticket to Databricks Enginee...
- 3 kudos
- 1399 Views
- 4 replies
- 0 kudos
Terraforming Git credentials for service principals
I am terraforming service principals in my Databricks workspace and it works great until I need to assign Git credentials to my SP. In the UI we have these options to configure credentials on service principal page:However the Terraform resource I fo...
- 1399 Views
- 4 replies
- 0 kudos
- 0 kudos
You're a little bit ahead of me in this process, so I haven't tried the solution yet, but it looks like you create a git credential resource for the service principal. This requires a token, which I think must be generated in the console. My refere...
- 0 kudos
- 1088 Views
- 1 replies
- 0 kudos
Unity Catalog: 403 Error When Connecting S3 via IAM Role and Storage Credential
Hi,We're currently setting up Databricks Unity Catalog on AWS. We created an S3 bucket and assigned an IAM role (databricks-storage-role) to give Databricks access.Note: Databricks doesn't use the IAM role directly. Instead, it requires a Storage Cre...
- 1088 Views
- 1 replies
- 0 kudos
- 0 kudos
Have you follow any specific guide for the creation of the same? Are you setting up a Unity Catalog Metastore or the default storage for the workspace?For the Metastore creation have you follow steps in https://docs.databricks.com/aws/en/data-governa...
- 0 kudos
