Explore discussions on Databricks administration, deployment strategies, and architectural best practices. Connect with administrators and architects to optimize your Databricks environment for performance, scalability, and security.
On Azure I added a service principal X to my databricks workspace. I therefore had the Service Prinicpal Manager role on that service principal X. I accidentally downgraded my rights to Service Principal User and now can's get my Managers role back. ...
I have a Databricks account with the E2 version of the Databricks platform. The unified login is not enabled on the account because it was created before June 21, 2023.I configured SSO authentication for the account and for a given workspace (separat...
Hi,I know that with single clusters, theres a single driver node and one driver per cluster. With shared clusters, multiple jobs can run concurrently. Does this still run on a single driver container or multiple driver containers run per application?...
I manage instance profiles assigned to specific user groups. For example, instance profile A provides access solely to group A. Currently, any user within group A has the ability to update the permissions of a cluster using instance profile A, which ...
Hi @Retired_mod,Thank you for your guidance. I am following the strategies outlined in steps 1 and 2, and I remain concerned about a specific scenario.Consider instance profile A, which is designed to grant access exclusively to group A. If user A, ...
I have numerous cluster policies varying that varies in sizes (Job - xsmall, Job - small, Job - medium...). However, when I create a new job and create a new job cluster the default policy selected from the drop down menu is on the bigger size. Is th...
Use case and context:We have a databricks workspace in a specific region, reading and writing files from/to the same region.We also read from a Shared Catalog in a different company, a data provider, which is pointing to multi-region s3 buckets.The r...
Thanks @Retired_mod for all the suggestions.After some days of monitoring NAT cost, I realized that the implementation of the S3 Gateway Endpoint it was actually working, the problem was that I thought that this change would be reflected right away i...
Hola all, I'm experiencing a quite strange error. The problem is that and happens inside a GITLAB pipeline:$ databricks current-user meError: default auth: cannot configure default credentials, please check https://docs.databricks.com/en/dev-tools/au...
Hola Kaniz, the problem is not on Databricks CLI but is due to some interactions happening inside the Gitlab pipeline. According to the documentation reported here: Databricks personal access token authentication | Databricks on AWS ( at the bottom o...
Currently, we have 3 Unity Catalog enabled workspaces sharing the same metastore. Now, when we create an external location or storage credential in any of the workspaces, it gets reflected across all workspaces. We are looking for some best practices...
Hi @Debi-Moha Currently we do not have a mechanism to isolate the external locations and storage credentials based on workspaces, since the metastore is shared across the workspaces.
Please check below document for recommendations on securing extern...
Hey,I am using Bitbucket Cloud and I want to connect my repository to Databricks. I am able to connect with my personal app password but what I am looking for is an authentication of a technical user.I need the integration to point to my dbt repo, wh...
Hi @Retired_mod,thank you for your response! With this link you provided, I was able to authenticate with Bitbucket Cloud. The solution was to use x-token-auth as a username. I have tried with the generated email address before which didn't work. Tha...
We want to build monitoring and Alerting solution for Azure Databricks that should capture Resource Utilization details (like Aggregated CPU%, Memory% etc.) and Cost consumption at the Account Level.We have Unity Catalog Enabled and there are multipl...
@smehta_0908 Greetings!
You can utilize Datadog for monitoring CPU and memory of clusters.
https://docs.datadoghq.com/integrations/databricks/?tab=driveronly
For Cost consumption at accounts level you can make use of billable usage logs using the Acc...
When experimenting with LLMs on Databricks clusters, I have become interested in knowing if the LLM (Llama2 or otherwise), tries to make calls to the internet (i.e., the settings for use_remote_code=True in Huggingface models, as just one example).Mo...
Hey @FlavioSM The error message indicates that the library you're trying to install (com.databricks:spark-xml_2.13:0.17.0) is not on the allowlist for shared clusters in your Databricks workspace. Shared clusters are clusters that multiple users can ...
Hi everyone,I would like to know if the following behavior is expected or if it is a misconfiguration in SCIMWe are going through a change in the email of some users. So we did a test, changing the email of one of them, but the result was not OK beca...
Hi @rfreitas , It is an expected behaviour because we don't support email or username updates/edits in the SCIM as of now. As per Databricks documentation, an email update is not supported in SCIM. You cannot update the username or email address of a...
Hi team,Greetings.Do you know if there is a way to update the creator of a databricks_job using the API? the Documentation does not show "the creator" property and when I tried setting the creator, this property is not updated in the workspace UI.The...
Hi @danmlopsmaz , Thanks for bringing up your concerns, always happy to help
I understand the customer wanted to change the creator of the job but at this moment we could change the owner but not the creator. But you should be able to clone an exist...
Is there a way to register a web app with Databricks to enable it to access a user's data using OAuth2? i.e., be included in the list of published apps with a pre-negotiated scope?
Join a Regional User Group to connect with local Databricks users. Events will be happening in your city, and you won’t want to miss the chance to attend and share knowledge.
If there isn’t a group near you, start one and help create a community that brings people together.