- 3829 Views
- 7 replies
- 3 kudos
Azure basic public IP SKU retirement
With Azure notifying that retirement of the basic public IP SKU in 2025 I'd like to understand how compute cluster workers can be moved over from basic to standard SKU? We've used Terraform to deploy our Databricks environment but I'd like to underst...
- 3829 Views
- 7 replies
- 3 kudos
- 3 kudos
I contacted Microsoft, they spoke with their SME of the Databricks team who have confirmed they're aware of this and will be transitioning from Basic to Standard SKU before deprecation - a update from Microsoft will be getting released regarding this...
- 3 kudos
- 2507 Views
- 4 replies
- 3 kudos
Resolved! Install Python dependency on job cluster from a privately hosted GitLab repository (HTTPS/SSH)
Hello,We intend to deploy a Databricks workflow based on a Python wheel file which needs to run on a job cluster. There is a dependency declared in pyproject.toml which is another Python project living in a private Gitlab repository. We therefore nee...
- 2507 Views
- 4 replies
- 3 kudos
- 3 kudos
Hi @andre-h ,As a good alternative you can build the python package (wheel or egg) in your gitlab or github workflows and upload the package to dedicated cloud storage bucket. Then followed by you can specify the cloud storage path of your python lib...
- 3 kudos
- 2116 Views
- 7 replies
- 2 kudos
Resolved! Unable to get CLI working on new laptop
Hi,I have two Windows 10 laptops. The old one is working fine with Databricks CLI version 0.210.0. I installed the CLI on a new laptop and copied my config file. I can test connections to my default config which uses my PAT. However, I have some ...
- 2116 Views
- 7 replies
- 2 kudos
- 2 kudos
Great to hear issue got resolved, let us know in case any other assistance is required.
- 2 kudos
- 1495 Views
- 4 replies
- 1 kudos
Resolved! regexp_count seems to not work as it should
The below SQL's should give different answers as regexp rules state that * is a special character which needs to be escaped with \ to be considered as a literal string. This second should literally match for A*B and return 2 but it is also taking AB ...
- 1495 Views
- 4 replies
- 1 kudos
- 1 kudos
Hello @Rajasql ,Try this way (it worked for me in a Databricks notebook and it returns 2):SELECT regexp_count('nA*BsABreA*Bthe', 'A\\*B') str_cnt;
- 1 kudos
- 781 Views
- 1 replies
- 1 kudos
Resolved! Compute fleets on Azure Databricks
It seems that compute fleets have been supported on AWS for almost 2 years. Azure compute fleets went into preview in November 2024. Has anyone heard of how or when compute fleets will be supported on Azure Databricks?
- 781 Views
- 1 replies
- 1 kudos
- 1 kudos
This is currently on development but to ETA has been shared yet by our engineering team. But might be coming soon.
- 1 kudos
- 2714 Views
- 11 replies
- 0 kudos
Can't create AWS p3 instance
Hi, I'm trying to create a `p3.2xlarge` in my workspace, but the cluster fails to instantiate, specifically getting this error message: `No zone supports both the driver instance type [p3.2xlarge] and the worker instance type [p3.2xlarge]` (though I ...
- 2714 Views
- 11 replies
- 0 kudos
- 0 kudos
I was able to start a cluster with same exact configuration on my internal environment with no issues, I have selected east-1a as the AZ to deploy.By any chance have you engaged AWS support on this?
- 0 kudos
- 1078 Views
- 3 replies
- 0 kudos
Mimic system table functionality at custom catalog level
Hi,I am exploring system tables . I want to have our environment specific data in different catalogs. While it is possible to get audit and other usage info from system tables under system catalog,how can I achieve the same in my custom catalog that ...
- 1078 Views
- 3 replies
- 0 kudos
- 0 kudos
Just to be clear, what you want to do is have a set of system tables such as audit logs in each catalog for your environments, so when you query the data for those tables you just get information from your environment. On this case there is no built ...
- 0 kudos
- 1652 Views
- 1 replies
- 1 kudos
Delta lake : delete data from storage manually instead of vacuum
Hi AllWe have a unique use case where we are unable to run vacuum to clean our storage space of delta lake tables. Since we have data partitioned by date, we plan to delete files older than a certain date directly from storage. Could this lead to any...
- 1652 Views
- 1 replies
- 1 kudos
- 1 kudos
Deleting files older than a certain date directly from storage without using the VACUUM command can lead to potential issues with your Delta Lake tables. Here are the key points to consider: Corruption Risk: Directly deleting files from storage can ...
- 1 kudos
- 1126 Views
- 1 replies
- 1 kudos
Resolved! Where is the Open Apache Hive Metastore API?
I 2023 it was announced that databricks has made a "Hive Metastore (HMS) interface for Databricks Unity Catalog, which allows any software compatible with Apache Hive to connect to Unity Catalog".Is this discontinued? If not, is there any documentati...
- 1126 Views
- 1 replies
- 1 kudos
- 1 kudos
It seems that this option has been deprecated, it was a private preview but is no longer available for enrollment
- 1 kudos
- 4416 Views
- 10 replies
- 0 kudos
Resolved! Failed to add 3 workers to the compute. Will attempt retry: true. Reason: Driver unresponsive
Currently I trying to Create a Compute Cluster on a Workspaces with Privatelink and Custom VPC.I'm using Terraform: https://registry.terraform.io/providers/databricks/databricks/latest/docs/guides/aws-private-link-workspaceAfter the deployment is com...
- 4416 Views
- 10 replies
- 0 kudos
- 0 kudos
Hi @ambigus9, Looks like based on connectivity test to the RDS it's not working. Can you check if there is any Firewall blocking the request, since connection is not going through the RDS.
- 0 kudos
- 1360 Views
- 1 replies
- 0 kudos
Resolved! How to Retrieve Admin and Non-Admin Permissions at Workspace Level in Azure Databricks.
Hello,I am working on a project to document permissions for both admins and non-admin users across all relevant objects at the workspace level in Azure Databricks (e.g., tables, jobs, clusters, etc.).I understand that admin-level permissions might be...
- 1360 Views
- 1 replies
- 0 kudos
- 0 kudos
In Databricks the object permissions are based in the object itself and not the user. Unfortunately as of now there is no way to get all the objects permissions in a single built in query.There is custom options as for example for clusters, first run...
- 0 kudos
- 2410 Views
- 2 replies
- 1 kudos
Resolved! Databricks Connect: Enabling Arrow on Serverless Compute
I recently upgraded my Databricks Connect version to 15.4 and got set up for Serverless, but ran into the following error when I ran the standard code to enable Arrow on Pyspark: >>> spark.conf.set(key='spark.sql.execution.arrow.pyspark.enabled', val...
- 2410 Views
- 2 replies
- 1 kudos
- 881 Views
- 3 replies
- 0 kudos
Can we change our cloud service connected with our Databricks account
We are moving from old aws account to azure account. Is there any way. I can migrate my old databricks account to this new azure account.I have my Databricks partner workspace access with this Databricks account. That's the reason, I want to keep thi...
- 881 Views
- 3 replies
- 0 kudos
- 0 kudos
Unfortunately I am not able to find a way to move the workspace, if you have an account representative within Databricks I will suggest to reach out to see any options you can have to migrate also this credits if possible
- 0 kudos
- 1823 Views
- 4 replies
- 0 kudos
Get a static IP for my Databricks App
Hello,I'm trying to find how to set-up a static IP for a Azure Databricks App. I tried to set-up a NAT gateway to have a static IP for the workspace, but it doesn't change anything, I still can't access my OpenAI ressource even if I authorize the NaT...
- 1823 Views
- 4 replies
- 0 kudos
- 0 kudos
Hi, I’m following up here as I have the same issue. Did the solution provided in the replies help resolve this for you?
- 0 kudos
- 972 Views
- 1 replies
- 1 kudos
Determining spill from system tables
I'm trying to optimize machine selection (D, E, or L types on Azure) for job clusters and all-purpose compute and am struggling to identify where performance is sagging on account of disk spill. Disk spill would suggest that more memory is needed. ...
- 972 Views
- 1 replies
- 1 kudos
- 1 kudos
For historical diagnostics, you might need to consider setting up a custom logging mechanism that captures these metrics over time and stores them in a persistent storage solution, such as a database or a logging service. This way, you can query and ...
- 1 kudos
