Hi everyone! I’m working on using cookiecutter to help us set up consistent project templates on Databricks. So far, it’s mostly working, but I’m struggling with the prompts – they’re not displaying well in the Databricks environment.I’ve tried some ...
Hi!I'd like to restrict some users' permissions using REST API and got an issue while trying to update a permission on 'directories'.I'm trying to set a user's permission on their default username folder in the workspace to 'can edit' so that they ca...
Hi @takak, Greetings from Databricks! What is the REST API you are making the call to? Looks like this might not be supported programmatically, but will try to test it internally. it appears that the CAN_MANAGE permission is a higher-level permission...
We are attempting to provision Unity Catalog using Terraform, but we're encountering issues with establishing authentication with AWS through IAM Roles and Policies.For EC2/Cluster instances, the instance profile works fine with a trust relationship ...
To provision Unity Catalog using Terraform and authenticate with AWS through IAM Roles and Policies, you'll need to follow these steps: Create an IAM Role for Unity Catalog: First, you need to create an IAM role that Unity Catalog can assume to acces...
We have a databricks job configured to run a dbt project. The dbt cli compute cluster being used is serverless with a serverless sql warehouse. We encountered this error during a run. SQLSTATE: XX000Any idea why this occurred?
Hello @Kutbuddin, The error you encountered, SQLSTATE: XX000 [INTERNAL_ERROR] Query could not be scheduled: HTTP Response code: 503. Please try again later, typically indicates an issue with the scheduling of the query due to a temporary unavailabili...
Is there an API that can be used to list groups in which a given user is a member? Specifically, I’d be interested in account (not workspace) groups.It seems there used to be a workspace-level list-parents API referred to in the answers to this quest...
@Alberto_Umana, do i understand correctly that there isn’t currently an endpoint that returns the info I need? May I request this feature be added on the roadmap? A bit of context: we have tens of catalogs in Unity Catalog and several workspaces in o...
#!/bin/bashpip install package1 --index-url https://link-to-indexpip install package2 --index-url https://link-to-indexThis init script fails witherror: externally-managed-environment× This environment is externally managed╰─> To install Python packa...
Hi @k1t3k, Are you installing a custom package? could you please share the package name you are installing to validate? The error you are encountering, "externally-managed-environment," when running your global init script with Databricks Runtime 16....
Is it possible to have the service principal (ID) of an Azure data factory as the Single user access on an databricks cluster ?Reason I'm asking is because we are starting to use unity catalog , but would still have the need to execute stored procedu...
Yes, this is possible. First, create a new service principal in Azure or use an existing one. This could be either a managed identity from Azure Data Factory or a manually created service principal in Microsoft Entra ID (formerly Azure AD). Next, in ...
How can I identify who triggered my Databricks job? The Databricks job is running via a service principal. One of my runs initially failed, but a repair occurred 30 minutes later, causing the job to enter a successful state. I would like to determine...
thanks for sharing the infocan you please share the audit logs query so I can pass.
I am facing an issue while adding a recipient to Delta Share using Terraform. The owner of my recipient is a group, not an individual user. I'm running this Terraform script using a service principal member of that group. However, I'm encountering th...
I was able to fix the issue. The problem was that the service principal I was using didn’t have the correct permissions assigned
When using notebooks with python in Databricks I would really like to easily see the defintion of the functions I am using within the editor. Which the "Go to definition (F12)" and "Peek definition" options when right clicking on the functions will h...
This has now been resolved and is working as expected. Do not know why or how, but something has changed that made it work
If we want to enable Databricks Predictive Optimization, then is it also mandatory to enable serverless Job/Notebook Compute in our account. We already have Serverless SQL warehouse available in our workspaces.
Thanks. This was answered in https://community.databricks.com/t5/administration-architecture/enable-predictive-optimization/td-p/98731
We run a shared cluster that is used for general purpose adhoc analytics, which I assume is a relatively common use case to try to keep costs down. However, the technical experience of users of this cluster varies a lot, so we run into situations whe...
Hi, @JameDavi_51481 , were you able to figure something out?Planning a Databricks migration and realized we might need something similar too.
Our SQL Serverless installation has sporadic failures to our blob container in Azure. The blob container is locked down to a vnet, and we are using the private endpoint to enable serverless access. It will work fine for several hours, and then show...
I've confirmed all of that. This seems like an AI generated response. It seems more likely that Databricks rolled out a feature a week ago that is causing instability in the serverless warehouses. Any other specific things to check would be apprec...
Privileged Identity Management (PIM) can be used to secure access to critical Databricks roles with Just-in-Time (JIT) access. This approach helps organizations enforce time-bound permissions, approval workflows, and centralized auditing for sensitiv...
Thanks. However, as what I know, Azure PIM does not work for Service Principals. It's only applied to human user access
"I want to create custom tags in a cluster policy so that clusters created using this policy will automatically include those tags for billing purposes. Consider the following example:"cluster_type": {"type": "fixed","value": "all-purpose"},"custom_t...
Are you having any issue while running this code in the policy?
