- 2428 Views
- 6 replies
- 9 kudos
Resolved! Is it possible restore a deleted catalog and schema
Is it possible restore a deleted catalog and schema.if CASCADE is used even though schemas and tables are present in catalog, catalog will be dropped.Is it possible to restore catalog or is possible to restrict the use of CACADE command.Thank you.
- 2428 Views
- 6 replies
- 9 kudos
- 9 kudos
@Louis_Frolio I cannot click any "Accept as Solution" button, as I was not the one creating the post, I believe
- 9 kudos
- 513 Views
- 1 replies
- 1 kudos
Resolved! Dev/Prod Environments in AWS: Separate Accounts vs. Separate Workspaces?
Hello everyone,I'm looking for some advice on best practices. When setting up development and production environments for Databricks on AWS, is it better to use completely separate AWS accounts, or is it sufficient to use separate workspaces within a...
- 513 Views
- 1 replies
- 1 kudos
- 1 kudos
Hi @tana_sakakimiya , I allow myself copy and paste brilliant answer on similar question provided by user Isi:"Option A: Multiple Databricks accounts and multiple AWS accountsThis model offers the highest level of isolation. Each environment lives in...
- 1 kudos
- 4576 Views
- 3 replies
- 4 kudos
Resolved! Enable the "Billable Usage Download' Accounts API on our Azure
Hi Databricks Support,Could you please confirm whether the Billable Usage Download endpoint can be enabled on Azure Databricks account, similar to how it’s available on AWS/GCP? If yes, what are the steps that should be followed for the same.. else ...
- 4576 Views
- 3 replies
- 4 kudos
- 3560 Views
- 1 replies
- 1 kudos
Mismatch cuda/cudnn version on Databricks Runtime GPU ML version
I have a cluster on Databricks with configuration Databricks Runtime Version16.4 LTS ML Beta (includes Apache Spark 3.5.2, GPU, Scala 2.12), and another cluster with configuration 16.0 ML (includes Apache Spark 3.5.2, GPU, Scala 2.12). According to...
- 3560 Views
- 1 replies
- 1 kudos
- 1 kudos
There could be library related conflicts in 16.0ML that got fixed in 16.4ML. I would always recommend to use the LTS version. Thanks
- 1 kudos
- 394 Views
- 2 replies
- 1 kudos
DataBricks JDBC driver fails when unsupported property is submitted
Hello!The Databricks jdbc driver applies a property that is not supported by the connector as a Spark server-side property for the client session. How can I avoid that ? With some tools I do not have 100% control, eg they may add a custom jdbc connec...
- 394 Views
- 2 replies
- 1 kudos
- 1 kudos
Hi @Kaitsu , The documentation mentions, If you specify a property that is not supported by the connector, then the connector attempts to apply the property as a Spark server-side property for the client session. Unlike many other JDBC drivers that ...
- 1 kudos
- 3992 Views
- 2 replies
- 2 kudos
Pre-loading docker images to cluster pool instances still requires docker URL at cluster creation
I am trying to pre-load a docker image to a Databricks cluster pool instance.As per this article I used the REST API to create the cluster pool and defined a custom Azure container registry as the source for the docker images.https://learn.microsoft....
- 3992 Views
- 2 replies
- 2 kudos
- 2 kudos
@NadithK Pre-loading Docker images to cluster pool instances is for performance optimization (faster cluster startup), but you still must specify the Docker image in your cluster configuration. The pre-loading doesn't eliminate the requirement to dec...
- 2 kudos
- 1248 Views
- 4 replies
- 3 kudos
Resolved! Databricks Default package repositories
I have added an extra-index-url in the default package repository in databricks which points to a repository in azure artifact. The libraries from it are getting installed on job cluster but is not working on the all purpose cluster. Below is the rel...
- 1248 Views
- 4 replies
- 3 kudos
- 3 kudos
Hello @tariq! Did the suggestions shared above help address your issue? If so, please consider marking one of the responses as the accepted solution. If you found a different approach that worked for you, sharing it with the community would be really...
- 3 kudos
- 1200 Views
- 1 replies
- 3 kudos
Resolved! Databricks Usage Dashboard - Tagging Networking Costs
Problem OverviewOur team has successfully integrated Azure Databricks Usage Dashboards to monitor platform-related costs. This addition has delivered valuable insights into our spending patterns. However, we've encountered a tagging issue that's prov...
- 1200 Views
- 1 replies
- 3 kudos
- 3 kudos
There is no direct way to tag certain Azure networking resources (such as network interfaces, public IPs, or managed disks) so that their costs inherit custom tags like "projectRole" in cost reports, because many core networking resources either do n...
- 3 kudos
- 417 Views
- 1 replies
- 0 kudos
Why MLFlow version blocks library installation?
I have a Databricks Asset Bundle proyect running and when the MLFlow library released the 3.4.0 version it happened this error:When restricting the MLFlow version to < 3.4.0 it works.The libraries got stuck installing and the process finished in erro...
- 417 Views
- 1 replies
- 0 kudos
- 0 kudos
Hello @PabloCSD! Which Databricks Runtime version are you using?
- 0 kudos
- 1737 Views
- 3 replies
- 3 kudos
Resolved! Markdown Cells Do Not Render Consistently
When I am creating a notebook in the UI editor on DataBricks, markdown cells do not always render after I run them. They still appear in 'editing mode'. See the screenshot below, it should have rendered a H1.Again, this behavior is not consistent. So...
- 1737 Views
- 3 replies
- 3 kudos
- 604 Views
- 2 replies
- 0 kudos
When setting up unity catalog a storage account was created with security risk
When i set up databricks unity catalog, i think it automatically set up a storage account. I'm getting recommendations from Azure that this storage account has high risk associated with it. The problem is this resource has a deny assignment on preven...
- 604 Views
- 2 replies
- 0 kudos
- 0 kudos
Hi @howardgagan ,Each Azure Databricks workspace has an associated Azure storage account in a managed resource group known as the workspace storage account.This storage account includes workspace system data (job output, system settings, and logs), D...
- 0 kudos
- 816 Views
- 2 replies
- 3 kudos
Any sample code snippet that contains end-to-end OIDC flow
Hello,Is there a sample code snippet that depicts end-to-end OIDC flow - imagine, there exists a service principal, interactive user who connect to an sql warehouse, get authenticated, and run some sql queries as part of a python script (jdbc/odbc) f...
- 816 Views
- 2 replies
- 3 kudos
- 3 kudos
Hello @noorbasha534, did the code snippet shared above help resolve your concern? If it did, please consider marking it as the accepted solution. If you found another approach, please share it with the community so others can benefit as well.
- 3 kudos
- 1323 Views
- 4 replies
- 1 kudos
Resolved! Spark Job fails with No plan for OptimizedForeachBatchFastpath
Hi Team,I am trying to run a job on Databricks cluster 14.3LTS which streams data from parquet to custom sink. I am facing below error. The same code used to work a month back, however i am facing this issue recently.org.apache.spark.SparkException: ...
- 1323 Views
- 4 replies
- 1 kudos
- 1 kudos
Disabling the setting worked for me on 16.4 (serverless):spark.databricks.streaming.forEachBatch.optimized.fastPath.enabled=false But you have to wonder if this is something Databricks are aware of and are looking to fix.
- 1 kudos
- 656 Views
- 1 replies
- 1 kudos
Resolved! Establish Cross cloud connectivity between Azure Databricks and AWS s3
Hello.We have the cross cloud configuration set as below:AWS - VPC, Transit Gateways, AWS Direct ConnectOn Premise Data centerAzure - Vnet, Transit Vnet , Express Route.We are trying to create a Databricks storage credential as below. The AWS IAM obj...
- 656 Views
- 1 replies
- 1 kudos
- 1 kudos
Hello @swee ,Thank you for your query.If your storage account is private, you would need to establish a route to that storage account so you can read data.This is because if your storage is private, your storage account will block access to the publi...
- 1 kudos
- 3885 Views
- 8 replies
- 2 kudos
Resolved! Reporting serverless costs to azure costs
So, we've just recently applied serverless budget polices to some of our vector searches and apps. At the moment they're all going to azure under one general tag that we created.However, we needed more definition. So i added the serverless budget pol...
- 3885 Views
- 8 replies
- 2 kudos
- 2 kudos
Billing or set up explicit export pipelines. Check whether your serverless budget policy tags are under a different namespace in Azure, as sometimes they show up nested.
- 2 kudos
