- 3929 Views
- 8 replies
- 3 kudos
Disable local user creation when using SCIM Provisioning
We have implemented SCIM Provisioning using Azure AD (MS Entra) to Azure Databricks.All is good.Except, we would like to know if it is possible to disable the ability to create users within Azure Databricks, so that none can be "accidentally" created...
- 3929 Views
- 8 replies
- 3 kudos
- 3 kudos
Thank you! Thats really clear now, and hopefully helpful to others.Ours is set to (default) OFF - we do not want JIT provisioning enabled.
- 3 kudos
- 4253 Views
- 1 replies
- 0 kudos
client.openSession() : TypeError: Cannot read properties of undefined (reading '0')
I am using the Databricks SQL Driver for Node.js to create an endpoint that queries a Databricks database following the guide here Databricks SQL Driver for Node.js | Databricks on AWS . This code was working previously but now I am getting a TypeErr...
- 4253 Views
- 1 replies
- 0 kudos
- 0 kudos
Your TypeError: Cannot read properties of undefined (reading '0') at session = await client.openSession() typically indicates an unexpected change or regression inside the Databricks SQL Node.js driver or the environment, even if your environment var...
- 0 kudos
- 3688 Views
- 2 replies
- 0 kudos
GCP Cluster will not boot correctly with Libraries preconfigured - notebooks never attach
I am running Databricks 15.4 LTS on a single-node `n1-highmem-32` for a PySpark / GraphFrames app (not using builtin `graphframes` on ML image because we don't need a GPU) and I can start the cluster fine so long as libraries are not attached. I can ...
- 3688 Views
- 2 replies
- 0 kudos
- 0 kudos
It sounds like you are encountering a cluster “hang”/notebook attach timeout after restarting a Databricks 15.4 LTS single-node cluster with custom libraries (including GraphFrames via Maven and additional .whl and requirements.txt dependencies). You...
- 0 kudos
- 3558 Views
- 2 replies
- 0 kudos
Asset Bundle: inject job start_time parameter
Hey!I'm deploying a job with databricks asset bundles.When the pyspark task is started on a job cluster, I want the python code to read the job start_time and select the right data sources based on that parameter.Ideally, I would read the parameter f...
- 3558 Views
- 2 replies
- 0 kudos
- 0 kudos
You cannot directly access a dynamic value like ${job.start_time.iso_datetime} in a Databricks Asset Bundle YAML for job parameters—Databricks jobs do not inject special variables (like the job run’s start time) automatically into job parameters at r...
- 0 kudos
- 492 Views
- 4 replies
- 4 kudos
Resolved! Connect to a SQL Server Database with Windows Authentication
Good Day all, I am in the process of trying to connect to one of our SQL servers. It is attached to our Entra for authentication. When trying to create an external connection to the Server in Unity I am getting a failure due to the User and Password ...
- 492 Views
- 4 replies
- 4 kudos
- 4 kudos
@Adam_Borlase Can you try this steps to see there is no network issue.Use SQL AuthenticationCreate a SQL Server login (not Entra ID) with a username and password.Grant it access to the required database.Use this credential in Unity Catalog's external...
- 4 kudos
- 4193 Views
- 1 replies
- 0 kudos
Adding service principal with Microsoft Entra ID fails
Hi,I am trying to add a service principal using Microsoft Entre ID, but I encounter an issue as described in the following documentation: https://learn.microsoft.com/en-us/azure/databricks/dev-tools/auth/oauth-m2m.I followed the instructions step by ...
- 4193 Views
- 1 replies
- 0 kudos
- 0 kudos
The error message you encountered—“Successfully created new service principal but failed to add the new service principal to this workspace. Error fetching user”—along with the service principal's absence in “Users,” typically points to a synchroniza...
- 0 kudos
- 3880 Views
- 2 replies
- 0 kudos
Budget Policy - Service Principals don't seem to be allowed to use budget policies
ObjectiveTransfer existing DLT pipeline to new owner (service principal). Budget policies enabled.Steps to reproduceCreated a service principalAssigned it group membership of a group that is allowed to use a budget policyEnsured it has access to the ...
- 3880 Views
- 2 replies
- 0 kudos
- 0 kudos
The error message "Pipeline 'Run As' identity does not have access to selected budget policy" typically indicates that, while your service principal is properly configured for general pipeline ownership, it’s missing explicit permission on the budget...
- 0 kudos
- 3774 Views
- 1 replies
- 1 kudos
Change AWS S3 storage class for subset of schema
I have a schema that has grown very large. There are mainly two types of tables in it. One of those types accounts for roughly 80% of the storage. Is there a way to somehow set a policy for those tables only to transfer them to a different storage cl...
- 3774 Views
- 1 replies
- 1 kudos
- 1 kudos
Yes, it's possible to manage storage costs in Databricks and Unity Catalog by targeting specific tables for different storage classes, but Unity Catalog does add complexity since it abstracts the direct S3 (or ADLS/GCS) object paths from you. Here’s ...
- 1 kudos
- 4893 Views
- 2 replies
- 3 kudos
Resolved! Create account group with terraform without account admin permissions
I’m trying to create an account-level group in Databricks using Terraform. When creating a group via the UI, it automatically becomes an account-level group that can be reused across workspaces. However, I’m struggling to achieve the same using Terra...
- 4893 Views
- 2 replies
- 3 kudos
- 3 kudos
You cannot create account-level groups in Databricks with Terraform unless your authentication mechanism has account admin privileges. This is a design limitation of both the Databricks API and Terraform provider, which require admin-level permission...
- 3 kudos
- 3426 Views
- 1 replies
- 0 kudos
"Break Glass" access for QA and PROD environments
We're a small team with three environments (development, qa, and production), each in a separate workspace. Our deployments are automated through CI/CD practices with manual approval gates to deploy to the qa and production environments.We'd like to ...
- 3426 Views
- 1 replies
- 0 kudos
- 0 kudos
Implementing "break glass" access control in Databricks, similar to Azure Privileged Identity Management (PIM), requires creating a process where users operate with minimal/default permissions, but can temporarily elevate their privileges for critica...
- 0 kudos
- 642 Views
- 1 replies
- 0 kudos
GKE Cluster Shows "Starting" Even After its turned on
Curious if anyone else has run into this. After changing to GKE based clusters, they all turn on but don't show as turned on - we'll have it show as "Starting" but be able to see the same cluster in the dropdown that's already active. "Changing" to t...
- 642 Views
- 1 replies
- 0 kudos
- 0 kudos
Yes, others have reported encountering this exact issue with Databricks clusters on Google Kubernetes Engine (GKE): after transitioning to GKE-based clusters, the UI may show clusters as "Starting" even though the cluster is already up and usable in ...
- 0 kudos
- 163 Views
- 1 replies
- 1 kudos
Reaching out to Azure Storage with IP from Private VNET pool
Hey All,Is there a way for Databricks to reach out to Azure Storage using private endpoint?We would like no omit enabling access by "all trusted services".All resources are in the same VNET however when Databrics tries to reach out to Storage instead...
- 163 Views
- 1 replies
- 1 kudos
- 1 kudos
Yeah, it’s definitely possible for Databricks to hit Azure Storage through a private endpoint without turning on “allow trusted services.” The key is making sure everything’s using the private network path.Right now, that 10.0.35.x IP you’re seeing i...
- 1 kudos
- 154 Views
- 1 replies
- 1 kudos
Power Automate Azure Databricks connector cannot get output result of a run
Hi everybody, I'm using the Azure Databricks connector in Power automate and try to trigger a job run + get result of that single run. My job created in databricks is to run a notebook that contains a single block of SQL code, and that's the only tas...
- 154 Views
- 1 replies
- 1 kudos
- 1 kudos
Even though your Databricks job only has one task, Power Automate might still treats it as a multi-task job under the hood. That’s why you're getting the error when trying to fetch the output directly from the job run.Here’s a simple workaround you c...
- 1 kudos
- 419 Views
- 5 replies
- 3 kudos
Resolved! Neeed help with setting up a connection from Databricks to an Azure SQL Database with the REST API
Good day,I need some help with automating a connection from databricks to an Azure SQL Database. I'am able to configure the connection with the UI (Catalog Explorer), but I also want to configure it with a REST API (or SQL script), so that I can inte...
- 419 Views
- 5 replies
- 3 kudos
- 3 kudos
Hi Bianca,Thanks for your help. If I understand correctly the "authorization_code" and "pkce_verifier" are normally generated by the button "Sign in with Azure Entra ID" when I configure a connection through the Catalog Explorer.My organization is ne...
- 3 kudos
- 3826 Views
- 1 replies
- 0 kudos
Using Databricks CLI for generating Notebooks not supported or not implemented
Hi I'm a Data engineer and recently developed a Notebook analytics template for general purposes that I would like to be the standard on my company. Continuing, I created another notebook with a text widget that uses the user input to map the folder ...
- 3826 Views
- 1 replies
- 0 kudos
- 0 kudos
The issue you’re facing is common among Databricks users who try to automate notebook cloning via shell commands or %sh magic, only to encounter format loss: exporting via %sh databricks workspace export or related commands typically results in .dbc,...
- 0 kudos
