- 1368 Views
- 1 replies
- 1 kudos
Delta Lake: Running Delete and writes concurrently
Is it safe to run a delete query when there are active writes to a delta lake table? Next question : Is it safe to run a vacuum when writes are being done actively?
- 1368 Views
- 1 replies
- 1 kudos
- 1 kudos
Hello @sharat_n ,Yes, it is generally safe to run a DELETE query on a Delta Lake table while active writes are happening.Delta Lake is designed with ACID transactions, meaning operations like DELETE, UPDATE, and MERGE are atomic and isolated.In other...
- 1 kudos
- 3338 Views
- 0 replies
- 0 kudos
Databricks report error: unexpected end of stream, read 0 bytes from 4 (socket was closed by server)
Has anyone encountered this error and knows how to resolve it?"Unexpected end of stream, read 0 bytes from 4 (socket was closed by server)."This occurs in Databricks while generating reports.I've already adjusted the wait_timeout to 28,800, and both ...
- 3338 Views
- 0 replies
- 0 kudos
- 1718 Views
- 2 replies
- 1 kudos
Resolved! Missing 'Permissions' settings for Delta Live Tables Pipelines
Context: Azure Databricks, I am account admin, workspace admin, and pipeline owner (as confirmed via the API and visually in the Pipelines screen). When attempting to grant CAN_MANAGE access to developers for our DLT pipelines via the Databricks web ...
- 1718 Views
- 2 replies
- 1 kudos
- 1 kudos
If you click on the link of the Pipeline and then you access the kebab menu you will see the Permissions page
- 1 kudos
- 997 Views
- 1 replies
- 0 kudos
Unable to create a databricks workspace
I am unable to create a databricks workspace with vnet injection. I get this error:│ { │ "status": "Failed", │ "error": { │ "code": "InternalServerError", │ "message": "INTERNAL_ERROR: Unexpected error: Cannot call getCertifiedMetastoreFo...
- 997 Views
- 1 replies
- 0 kudos
- 0 kudos
The issue resolved itself when I tried to create a new resource group, dedicated to just Databricks. I don't know why that worked. If anyone know what went wrong, I would appreciate feedback!
- 0 kudos
- 6374 Views
- 5 replies
- 1 kudos
Resolved! Unable to access Live Ganglia UI
I’m having a trouble accessing Ganglia UI on a job run cluster. The job run is executed by a service principal and I would like to enable the Ganglia UI view to an user/admin group in Databricks. The error I get is: HTTP ERROR 403 PERMISSION_DENIED: ...
- 6374 Views
- 5 replies
- 1 kudos
- 1 kudos
Hi @arkadiuszr - can you please try the following steps and let us know? To resolve this error, first you want to add your service principal to the workspace: Go to admin console as a workspace adminSelect “service principal tab”Click “Add service ...
- 1 kudos
- 3295 Views
- 0 replies
- 0 kudos
VPAT Form
How do I find a Voluntary Product Accessibility Template (VPAT) from Databricks?
- 3295 Views
- 0 replies
- 0 kudos
- 6146 Views
- 7 replies
- 5 kudos
Resolved! How to get rid of a pesky gen AI feature in the editor ?
Hi,The editor interface has that gen AI feature following empty lines with a cursor. I find that very distracting and irritating. More over, once a line is deleted that unsolicited thing is interfering with code (snapshots included).How to get rid of...
- 6146 Views
- 7 replies
- 5 kudos
- 5 kudos
Summary:I was not able to solve the UI/UX artifact on my own (on user side)The UI/UX issue was resolved somewhere on Databricks side. The UI/AX artifact is no longer interfering with work.
- 5 kudos
- 895 Views
- 1 replies
- 0 kudos
Resolved! Attach a databricks_instance_pool to databricks_cluster_policy via terraform
Hello Team,I am trying to create a databricks instance pool and attach it to a cluster policy in our terraform code. But I am having hard time finding a good documentation. Has any one done it? Below is my sample code and I am getting errorI keep get...
- 895 Views
- 1 replies
- 0 kudos
- 0 kudos
Fixed it! "instance_pool_id" : { type = "fixed" values = "databricks_instance_pool.dev_test_cluster_pool.id"}
- 0 kudos
- 1111 Views
- 2 replies
- 0 kudos
Lakehouse federation -' on behalf of ' queries
Is it possible to achieve the following in a lake-house federation setup using Azure Databricks?1. Establish an external connection (EC1) to an external data source (EDS) using the credentials of user U1.2. Create a foreign catalog (FC1) utilizing EC...
- 1111 Views
- 2 replies
- 0 kudos
- 0 kudos
Thanks for explaining the authorization flow, @rangu . In the example mentioned, does Databricks support passing the user’s credentials to an external data source? For instance, can it pass the OAuth token for the user along with the externalID crede...
- 0 kudos
- 3412 Views
- 0 replies
- 0 kudos
GitLab on DCS, Datarbricks Container Services
I would like to set up GitLab and Grafana servers using Databricks Container Services (DCS). The reason is that our development team is small, and the management costs of using EKS are not justifiable. We want to make GitLab and Grafana accessible in...
- 3412 Views
- 0 replies
- 0 kudos
- 2148 Views
- 4 replies
- 0 kudos
Libraries installation governance
Dear allI like to know the best practices around libraries installation on Databricks compute - all-purpose, job.The need is to screen the libraries, conduct vulnerability tests, and then let them be installed through a centralized CI/CD process. How...
- 2148 Views
- 4 replies
- 0 kudos
- 0 kudos
@filipniziol thanks again for your time. The thing is we like to block access to these URLs as at times we found developers & data scientists downloading packages that were marked as vulnerable by Maven.
- 0 kudos
- 1138 Views
- 2 replies
- 0 kudos
Azure Databricks Serverless Compute
Hello,Looking for documents related to Azure Databricks Serverless Compute. What are the things we need to consider for security point of view when we decide to use serverless compute?
- 1138 Views
- 2 replies
- 0 kudos
- 0 kudos
These steps are really helpful. I especially appreciate the reminder to check my credentials and consider browser-related issues, as those are often overlooked. I'll make sure to clear my cache and cookies first, and if that doesn't work, I’ll try us...
- 0 kudos
- 831 Views
- 1 replies
- 0 kudos
How to NOT install or disable or uninstall Databricks Delta Live Tables dlt module on jobs cluster?
I need to NOT have the Databricks Delta Live Tables (DLT) Python stub installed on job cluster b/c of naming conflict w/ pip library dlt (and I also don't need delta live tables).There is no "simple" way of uninstalling. It's not installed via pip as...
- 831 Views
- 1 replies
- 0 kudos
- 0 kudos
For anyone facing a similar problem: I've addressed the issue of the dlt module conflict on my job cluster, by using an init script to remove the dlt module from the cluster's Python environment.Simply by doing:%bash #!/bin/bash rm -rf /databricks/sp...
- 0 kudos
- 1025 Views
- 1 replies
- 0 kudos
legacy repo error fetching git status files over 200MB
Working directory contains files that exceed the allowed limit of 200 MB. how to solve this?
- 1025 Views
- 1 replies
- 0 kudos
- 0 kudos
Hi @JessieWen ,What you can do besides removing some files from the repo, is to use "Sparce mode" and select only certain paths to be synchronized with Databricks repos. Hope it helps
- 0 kudos
- 6480 Views
- 8 replies
- 0 kudos
Databricks SQL connectivity in Python with Service Principals
Tried to use M2M OAuth connectivity on Databricks SQL Warehouse in Python:from databricks.sdk.core import Config, oauth_service_principal from databricks import sql .... config = Config(host=f"https://{host}", client_...
- 6480 Views
- 8 replies
- 0 kudos
- 0 kudos
Did anyone get this to work? I have tried the code above but I get a slightly different error but I don't see the same level of details from the logs2024-10-04 14:59:25,508 [databricks.sdk][DEBUG] Attempting to configure auth: pat2024-10-04 14:59:25,...
- 0 kudos
