- 6135 Views
- 5 replies
- 0 kudos
Azure Entra SSO Error: Your user has not been registered
I have set up SSO within databricks and automatic user provisioning with Azure Entra and confirmed it is working for all users. However 1 user is presented with this when signing in. The user is in the enterprise app within Azure Entra and the user i...
- 6135 Views
- 5 replies
- 0 kudos
- 2561 Views
- 0 replies
- 1 kudos
Databricks Clean Room costing
hi,Can you throw some light on how the compute, data sharing costing is done for various scenarios:1. Collaborator 1 and Collaborator 2 are having Databricks accounts in the same region and same cloud. Is there a DBU cost and who will pay for it? I a...
- 2561 Views
- 0 replies
- 1 kudos
- 2151 Views
- 2 replies
- 0 kudos
Resolved! Databricks Spot Instance: Completion Guarantee
Databricks allows to use spot instances for worker nodes. I consider to use them for interactive clusters. Do I have a gurantee that code will be completed without any errors even if spot instances are evicted? I would accept execution delays but no ...
- 2151 Views
- 2 replies
- 0 kudos
- 0 kudos
You could explore their "SPOT_WITH_FALLBAK" feature. If you don't want your jobs to fail because of eviction but this currently is not supported with interactive clusters. Hoping that they may extend this to all compute options soonCreate a pipeline ...
- 0 kudos
- 6403 Views
- 4 replies
- 0 kudos
Show all privileges granted to principal
Given the name of a principal in Databricks (I'm using account-level groups) is there an easy way to query or in other way obtain all privileges granted to this principal?I know I can obtain the information by querying in several of the system.inform...
- 6403 Views
- 4 replies
- 0 kudos
- 0 kudos
This link will provide details on how to verify all the privileges granted to Service Principals
- 0 kudos
- 1833 Views
- 1 replies
- 1 kudos
Databricks Cache Options
Hi,We are working on Databricks solution hosted on AWS. We are exploring the caching options in Databricks. Apart from the Databricks cache and spark cache? What are the options? Is it feasible to use 3rd party Cache solutions like AWS Elastic Cache ...
- 1833 Views
- 1 replies
- 1 kudos
- 1 kudos
Databricks provides several caching options to enhance performance by minimizing Input and Output (I/O) read and write operations. These include: Databricks Disk Cache: This cache accelerates data reads by creating copies of remote Parquet data file...
- 1 kudos
- 2990 Views
- 3 replies
- 0 kudos
Resolved! How could we share the Databricks ML runtime cluster among users when enable Unity Catalog
Hi team,Currently, we use the Databricks ML runtime to run our workflows and sometimes do the EDA. What we need is that we want to create a Databricks ML runtime for the team to share. When enabling Unity Catalog, how could we create a shared ML runt...
- 2990 Views
- 3 replies
- 0 kudos
- 0 kudos
Right now there is not plan to support ML runtime in shared clusters. Engineering is working on additional solutions but no ETA is currently available.In regards why it is not supported, principal reason is due to isolation which is not available in ...
- 0 kudos
- 1108 Views
- 0 replies
- 1 kudos
Delta live table : run_as
Does Databricks have any plans to decouple the owner from the "run_as" identity in Delta Live Table like it can be done in jobs?The problem arise specially when using DABs. The service principal used to deploy DLTs shouldn't be the owner AND the runn...
- 1108 Views
- 0 replies
- 1 kudos
- 3655 Views
- 0 replies
- 0 kudos
Pass secret in spark config when value is in form a.b.c={{secrets/scope/secret}}
I am configuring the Cluster for a spark-submit task and I am trying to specify `spark.executor.extraJavaOptions a.b.c={{secrets/scope/secret}}` but the literal {{secrets/scope/secret}} is being passed in rather than the secret value itself.I know th...
- 3655 Views
- 0 replies
- 0 kudos
- 1427 Views
- 1 replies
- 0 kudos
Updating python version from 3.8 to 3.12 for s3 ingestion
I went to AWS Cloudformation stack and edited the template from python 3.8 to 3.12 and updated. I did this for both the workspace stack and the s3 ingestion stack. Will it break anything? Do I need to make any changes in the python code in the templa...
- 1427 Views
- 1 replies
- 0 kudos
- 0 kudos
Hi @Retired_mod ,Thanks a lot! I will look at StackSets.As I mentioned the code is not written by me but Databricks. Why does Databricks not use a new python version in its default stacks? We are low on resources and heavily rely on the default Datab...
- 0 kudos
- 3058 Views
- 3 replies
- 5 kudos
Resolved! Move to 100% Serverless
Hi all,A few questions about the upcoming transition to 100% serverless, if anyone has any info that would be great!When will the move to serverless occur? I understand from 1st July (today) but has anyone seen a roadmap?What will the move to serverl...
- 3058 Views
- 3 replies
- 5 kudos
- 5 kudos
Hi, so our Databricks contact just assured us the following, after we asked about this issue:Databricks is officially (but won’t be GA in every region till end of July) 100% Serverless OPTIONAL.We understand many of our customers have begged for 100%...
- 5 kudos
- 1643 Views
- 1 replies
- 0 kudos
Resolved! Unity Catalog Pandas on Spark Limitation
According to Databricks UC Documentation, below are the some of the limitations on Shared Mode Cluster.1. In Databricks Runtime 13.3 LTS and above, Python scalar UDFs and Pandas UDFs are supported. Other Python UDFs, including UDAFs, UDTFs, and Panda...
- 1643 Views
- 1 replies
- 0 kudos
- 0 kudos
@RamlaSuhra, UDFs on Unity Catalog is a feature that, at the current moment is still on the Public Preview stage. This means that the development has yet not finished. UDFs can be used on DBR 13.3 and above, UDAFs are already available for DBR 15.2 ...
- 0 kudos
- 1069 Views
- 2 replies
- 0 kudos
New Users don't receive onboarding email
When I create a new user in Databricks, the new user does not receive their onboarding email. It is not in their junkmail, deleted items or in their inbox.However, when I reset that user's password, they do receive the password reset link, and are a...
- 1069 Views
- 2 replies
- 0 kudos
- 0 kudos
Hi @billraper , I'm sorry to hear about the trouble. Would you mind sharing more about whether this is happening with a community portal profile or with a product profile? Please share the link to the profile with which you're experiencing this issue...
- 0 kudos
- 807 Views
- 1 replies
- 0 kudos
cannot login to account management
HiI am not able to login to account management (https://accounts.cloud.databricks.com), It somehow reninforce SSO, cannot login with username and password.
- 807 Views
- 1 replies
- 0 kudos
- 1590 Views
- 1 replies
- 1 kudos
Coarse-grained access management for jobs
Are there any perspectives in Databricks' roadmap for enabling coarse-grained access management for jobs?Currently, access to jobs has to be managed on a job by job basis: https://docs.databricks.com/en/security/auth-authz/access-control/index.html#j...
- 1590 Views
- 1 replies
- 1 kudos
- 1 kudos
Hi @Retired_mod, thanks for your reply.A more mature access management concept in Databricks would be definitely terrific. I understand it's not entirely along the AI-lines that Databricks is pushing hard currently, but it would improve a lot the pla...
- 1 kudos
- 23441 Views
- 8 replies
- 3 kudos
Resolved! Run workflow using git integration with service principal
Hi,I want to run a dbt workflow task and would like to use the git integration for that. Using my personal user I am able to do so but I am running my workflows using a service principal.I added git credentials and the repository using terraform. I a...
- 23441 Views
- 8 replies
- 3 kudos
- 3 kudos
I created that link using the "Share" button in the post but it's broken, sorry Here's a working link to the discussion: https://community.databricks.com/t5/data-engineering/git-credentials-for-service-principals-running-jobs/td-p/73802
- 3 kudos
