- 358 Views
- 3 replies
- 0 kudos
how to restrict creation serving endpoints in databricks to a user
Is it possible somehow to restrict creation of the serving endpoints to specific users? I want to grant Workspace access under the Entitlements of the specific group, but I want not to allow users of this group create serving endpoints.The only way I...
- 358 Views
- 3 replies
- 0 kudos
- 0 kudos
@drag7ter While I mentioned Workspace-level entitlements as an approach, I need to clarify something important: Databricks doesn't currently support creating fully customized entitlements where you can exclude specific permissions like "serving endpo...
- 0 kudos
- 278 Views
- 1 replies
- 0 kudos
How to get response from API call made via executor
Hi Guys,I'm trying to call multiple APIs via executor using foreach partition, However as API response is getting returned at executor level I'm unable to see the response of API weather its 200 or 500.I dont want my APIs to execute on driver so I'm ...
- 278 Views
- 1 replies
- 0 kudos
- 0 kudos
Vaibhavaher2025 - I recommend trying the following: 1. Write logs from executors to persist storage insideprocess_partition. 2. Use mapPartitions instead offoreachPartition to return responses back to the driver as a Dataframe 3. Check executor log...
- 0 kudos
- 20080 Views
- 14 replies
- 3 kudos
Resolved! Databricks SQL - Unable to Escape Dollar Sign ($) in Column Name
It seems that due to how Databricks processes SQL cells, it's impossible to escape the $ when it comes to a column name.I would expect the following to work:%sql SELECT 'hi' `$id`The backticks ought to escape everything. And indeed that's exactly wha...
- 20080 Views
- 14 replies
- 3 kudos
- 541 Views
- 6 replies
- 0 kudos
Resolved! Serverless: ERROR: Could not install packages due to an OSError: [Errno 13] Permission denied
Hello guys!I am getting this error when running a job:ERROR: Could not install packages due to an OSError: [Errno 13] Permission denied: '/local_disk0/.ephemeral_nfs/cluster_libraries/python/lib/python3.11/site-packages/some-python-package'I have lis...
- 541 Views
- 6 replies
- 0 kudos
- 3737 Views
- 22 replies
- 1 kudos
Resolved! BQ partition data deleted fully even though 'spark.sql.sources.partitionOverwriteMode' is DYNAMIC
We have a date (DD/MM/YYYY) partitioned BQ table. We want to update a specific partition data in 'overwrite' mode using PySpark. So to do this, I applied 'spark.sql.sources.partitionOverwriteMode' to 'DYNAMIC' as per the spark bq connector documentat...
- 3737 Views
- 22 replies
- 1 kudos
- 1 kudos
@soumiknow , Just checking if there are any further questions, and did my last comment help?
- 1 kudos
- 282 Views
- 2 replies
- 2 kudos
Dataframe is getting empty during execution of daily job with random pattern
Hello, I have a daily ETL job that adds new records to a table for the previous day. However, from time to time, it does not produce any output.After investigating, I discovered that one table is sometimes loaded as empty during execution. As a resul...
- 282 Views
- 2 replies
- 2 kudos
- 2 kudos
Thank you very much, @BigRoux , for such a detailed and insightful answer!All tables used in this processing are managed Delta tables loaded through Unity Catalog.I will try running it with spark.databricks.io.cache.enabled set to false just to see i...
- 2 kudos
- 646 Views
- 6 replies
- 4 kudos
Databricks warehouse table optimization
Hi everyone,I just started using the Databricks and wanted to evaluate the reading speeds when using the Databricks warehouse.So I've generated the dataset of 100M records, which contains name, surname, date of birth, phone number and an address. Dat...
- 646 Views
- 6 replies
- 4 kudos
- 4 kudos
Hi Brahmareddy and AndrewN,Thank you on your answers.I first need to apologize as I accidentally wrote wrong that I got 270ms with hashing the date of birth, surname and name and then using the z ordering.I actually achieved around 290ms with hashing...
- 4 kudos
- 324 Views
- 4 replies
- 0 kudos
API Definition File
Hello. We are in the process of setting up Azure APIM to Databricks Rest API(s). Is there an official definition file available for download?Any help would be greatly appreciated
- 324 Views
- 4 replies
- 0 kudos
- 0 kudos
Thank you for the feedback. The postman collection would be ideal but the link is a no longer active
- 0 kudos
- 476 Views
- 1 replies
- 2 kudos
Issue pulling Docker Image on Databricks Cluster through Azure Container Registry
Hi Community.Essentially, we're using the ACR to push our custom Docker Image, then we would like to pull it to create a Databricks cluster. However, during the cluster creation, we got the following error:I'm convinced we tried to authenticate in al...
- 476 Views
- 1 replies
- 2 kudos
- 2 kudos
Anyone knows the cause issue?
- 2 kudos
- 177 Views
- 1 replies
- 0 kudos
Unable to create a metastore for Unity Catalog as I don't have Account Admin rights
Hello guys,I just started learning Databricks. I created a Databricks workspace via the Azure Portal using the Trial (Premium - 14-Days Free DBUs) plan. The workspace name is `easewithdata-adb`.However,I do not currently see the option to create a Un...
- 177 Views
- 1 replies
- 0 kudos
- 0 kudos
Hi @harika5991 You're right about the root cause of your issue. Creating a Unity Catalog metastore requires Account Admin privileges, which is separate from just creating a workspace in Azure.These are options you can try:When you create a Databricks...
- 0 kudos
- 462 Views
- 4 replies
- 4 kudos
Resolved! What are your most impactful use cases for schema evolution in Databricks?
Data Engineers, Share Your Experiences with Delta Lake Schema Evolution! We're calling on all data engineers to share their experiences with the powerful schema evolution feature in Delta Lake. This feature allows for seamless adaptation to changin...
- 462 Views
- 4 replies
- 4 kudos
- 231 Views
- 1 replies
- 0 kudos
Resolved! ConcurrentDeleteDeleteException in liquid cluster table
I am doing a merge in a table in parallel via 2 jobs.The table is a liquid clustered table with the following properties:delta.enableChangeDataFeed=truedelta.enableDeletionVectors=truedelta.enableRowTracking=truedelta.feature.changeDataFeed=supported...
- 231 Views
- 1 replies
- 0 kudos
- 0 kudos
Hey @flashmav , keep in mind that operations in Delta Lake often occur at the file level rather than the row level. For example, if two sessions attempt to update data in the same file (even if they’re not updating the same row), you may encounter a...
- 0 kudos
- 6727 Views
- 6 replies
- 6 kudos
Issue with Docker Image connection
Hello, I have created and pushed a docker image to Azure Container Registry . I used that image to start the cluster in Databricks. But the cluster doesn't start and as such when I try to run a Databricks' Jon using that Cluster, I get this error bel...
- 6727 Views
- 6 replies
- 6 kudos
- 7721 Views
- 6 replies
- 4 kudos
DLT | Cluster terminated by System-User | INTERNAL_ERROR: Communication lost with driver. Cluster 0312-140502-k9monrjc was not reachable for 120 seconds
Dear Community, Hope you are doing well.For the last couple of days I am seeing very strange issues with my DLT pipeline, So every 60-70 mins it is getting failed in continuous mode, with the ERROR; INTERNAL_ERROR: Communication lost with driver. Clu...
- 7721 Views
- 6 replies
- 4 kudos
- 4 kudos
We had similar error for one the DLT pipeline, This could be some times because of compute size, we had increased compute size of server in your DLT pipelines, still we were seeing this error while processing very large file. we then added below para...
- 4 kudos
- 572 Views
- 3 replies
- 1 kudos
Exporting table to GCS bucket using job
Hi all,Usecase: I want to send the result of a query to GCS bucket location in json format.Approach: From my java based application I create a job and that job will be running a notebook`. Notebook will have something like this```query = "SELECT * FR...
- 572 Views
- 3 replies
- 1 kudos
| User | Count |
|---|---|
| 1612 | |
| 768 | |
| 348 | |
| 286 | |
| 252 |
