cancel
Showing results for 
Search instead for 
Did you mean: 
Administration & Architecture
Explore discussions on Databricks administration, deployment strategies, and architectural best practices. Connect with administrators and architects to optimize your Databricks environment for performance, scalability, and security.
cancel
Showing results for 
Search instead for 
Did you mean: 
Data + AI Summit 2024 - Data Lakehouse Architecture


Forum Posts

ShankarM
by New Contributor III
  • 690 Views
  • 1 replies
  • 0 kudos

Resolved! Databricks Clean Room costing

hi,Can you throw some light on how the compute, data sharing costing is done for various scenarios:1. Collaborator 1 and Collaborator 2 are having Databricks accounts in the same region and same cloud. Is there a DBU cost and who will pay for it? I a...

  • 690 Views
  • 1 replies
  • 0 kudos
Latest Reply
Kaniz_Fatma
Community Manager
  • 0 kudos

Hi @ShankarM,  Collaborators in the same region and cloud: If both Collaborator 1 and Collaborator 2 have Databricks accounts in the same region and cloud, they will incur DBU (Databricks Unit) costs based on their usage. DBUs represent compute resou...

  • 0 kudos
VJ3
by New Contributor III
  • 801 Views
  • 2 replies
  • 1 kudos

Security Consideration for OAUTH Secrets to use Service Principal to authenticate with Databricks

What are the security consideration we need to keep in mind when we want to us OAUTH Secrets to use a Service Principal to access Azure Databricks when Identity federation is disabled and workspace is not yet on boarded on to Unity Catalog? Can we co...

  • 801 Views
  • 2 replies
  • 1 kudos
Latest Reply
VJ3
New Contributor III
  • 1 kudos

Thank you @Kaniz_Fatma for the response. I do have follow up questions.- What kind of encryption is used to store OAUTH secret?-  Is there any way OAUTH can be generated by someone else who is not a manager of that SPN? We need this as a part of segr...

  • 1 kudos
1 More Replies
a_user12
by New Contributor II
  • 1333 Views
  • 2 replies
  • 0 kudos

Resolved! Databricks Spot Instance: Completion Guarantee

Databricks allows to use spot instances for worker nodes. I consider to use them for interactive clusters. Do I have a gurantee that code will be completed without any errors even if spot instances are evicted? I would accept execution delays but no ...

a_user12_5-1719901164567.png
  • 1333 Views
  • 2 replies
  • 0 kudos
Latest Reply
imsabarinath
New Contributor III
  • 0 kudos

You could explore their "SPOT_WITH_FALLBAK" feature. If you don't want your jobs to fail because of eviction but this currently is not supported with interactive clusters. Hoping that they may extend this to all compute options soonCreate a pipeline ...

  • 0 kudos
1 More Replies
erigaud
by Honored Contributor
  • 13058 Views
  • 9 replies
  • 9 kudos

Resolved! Installing libraries on job clusters

Simple question : what is the way to go to install libraries on job clusters ? There does not seem to be a "Libraries" tab on the UI as opposed to regular clusters. Does it mean that the only option is to use init scripts ? 

  • 13058 Views
  • 9 replies
  • 9 kudos
Latest Reply
imsabarinath
New Contributor III
  • 9 kudos

You may want to copy required libs to a volume and load it during cluster setup to avoid downloading the libs for every run.

  • 9 kudos
8 More Replies
macmiller1
by New Contributor II
  • 3179 Views
  • 1 replies
  • 2 kudos

Resolved! Pass secret in spark config when value is in form a.b.c={{secrets/scope/secret}}

I am configuring the Cluster for a spark-submit task and I am trying to specify `spark.executor.extraJavaOptions a.b.c={{secrets/scope/secret}}` but the literal {{secrets/scope/secret}} is being passed in rather than the secret value itself.I know th...

  • 3179 Views
  • 1 replies
  • 2 kudos
Latest Reply
Kaniz_Fatma
Community Manager
  • 2 kudos

Hi @macmiller1,  Instead of using the {{secrets/scope/secret}} syntax, you can try using environment variables.If you prefer to use the {{secrets/scope/secret}} syntax, you can try escaping the equal sign (=) in your value.One way to do this is by us...

  • 2 kudos
KevinGagnon
by New Contributor
  • 569 Views
  • 1 replies
  • 0 kudos

Delta live table : run_as

Does Databricks have any plans to decouple the owner from the "run_as" identity in Delta Live Table like it can be done in jobs?The problem arise specially when using DABs. The service principal used to deploy DLTs shouldn't be the owner AND the runn...

  • 569 Views
  • 1 replies
  • 0 kudos
Latest Reply
Kaniz_Fatma
Community Manager
  • 0 kudos

Hi @KevinGagnon, Databricks currently does not have plans to decouple the owner from the "run_as" identity in Delta Live Tables, unlike what can be done with jobs.  The key points are: The Delta Live Table pipeline runs using the credentials of the p...

  • 0 kudos
alm
by New Contributor III
  • 3387 Views
  • 5 replies
  • 0 kudos

Show all privileges granted to principal

Given the name of a principal in Databricks (I'm using account-level groups) is there an easy way to query or in other way obtain all privileges granted to this principal?I know I can obtain the information by querying in several of the system.inform...

  • 3387 Views
  • 5 replies
  • 0 kudos
Latest Reply
sakthi_sujitha
New Contributor II
  • 0 kudos

This link will provide details on how to verify all the privileges granted to Service Principals 

  • 0 kudos
4 More Replies
sk_databricks
by New Contributor
  • 435 Views
  • 1 replies
  • 0 kudos

Databricks Cache Options

Hi,We are working on Databricks solution hosted on AWS. We are exploring the caching options in Databricks. Apart from the Databricks cache and spark cache? What are the options? Is it feasible to use 3rd party Cache solutions like AWS Elastic Cache ...

  • 435 Views
  • 1 replies
  • 0 kudos
Latest Reply
Walter_C
Honored Contributor
  • 0 kudos

Databricks provides several caching options to enhance performance by minimizing Input and Output (I/O) read and write operations. These include: Databricks Disk Cache: This cache accelerates data reads by creating copies of remote Parquet data file...

  • 0 kudos
ccsong
by New Contributor II
  • 886 Views
  • 3 replies
  • 0 kudos

How could we share the Databricks ML runtime cluster among users when enable Unity Catalog

Hi team,Currently, we use the Databricks ML runtime to run our workflows and sometimes do the EDA. What we need is that we want to create a Databricks ML runtime for the team to share. When enabling Unity Catalog, how could we create a shared ML runt...

  • 886 Views
  • 3 replies
  • 0 kudos
Latest Reply
Walter_C
Honored Contributor
  • 0 kudos

Right now there is not plan to support ML runtime in shared clusters. Engineering is working on additional solutions but no ETA is currently available.In regards why it is not supported, principal reason is due to isolation which is not available in ...

  • 0 kudos
2 More Replies
sandipkumar
by New Contributor II
  • 698 Views
  • 2 replies
  • 0 kudos

Updating python version from 3.8 to 3.12 for s3 ingestion

I went to AWS Cloudformation stack and edited the template from python 3.8 to 3.12 and updated. I did this for both the workspace stack and the s3 ingestion stack. Will it break anything? Do I need to make any changes in the python code in the templa...

  • 698 Views
  • 2 replies
  • 0 kudos
Latest Reply
sandipkumar
New Contributor II
  • 0 kudos

Hi @Kaniz_Fatma ,Thanks a lot! I will look at StackSets.As I mentioned the code is not written by me but Databricks. Why does Databricks not use a new python version in its default stacks? We are low on resources and heavily rely on the default Datab...

  • 0 kudos
1 More Replies
dbrx_user
by New Contributor III
  • 1328 Views
  • 4 replies
  • 6 kudos

Resolved! Move to 100% Serverless

Hi all,A few questions about the upcoming transition to 100% serverless, if anyone has any info that would be great!When will the move to serverless occur? I understand from 1st July (today) but has anyone seen a roadmap?What will the move to serverl...

  • 1328 Views
  • 4 replies
  • 6 kudos
Latest Reply
m997al
Contributor III
  • 6 kudos

Hi, so our Databricks contact just assured us the following, after we asked about this issue:Databricks is officially (but won’t be GA in every region till end of July) 100% Serverless OPTIONAL.We understand many of our customers have begged for 100%...

  • 6 kudos
3 More Replies
Sadam97
by New Contributor
  • 571 Views
  • 1 replies
  • 0 kudos

Databricks (GCP) Cluster not resolving Hostname into IP address

we have #mongodb hosts that must be resolved to private internal loadbalancer ips ( of another cluster ), and that we are unable to add host aliases in the Databricks GKE cluster in order for the spark to be able to connect to a mongodb and resolve t...

  • 571 Views
  • 1 replies
  • 0 kudos
Latest Reply
Kaniz_Fatma
Community Manager
  • 0 kudos

Hi @Sadam97,  First, verify if your DNS server is responding. You can do this by running a ping command from a notebook in Databricks to reach your secondary DNS server.Launch a Web Terminal from the cluster workspace.Edit the /etc/resolv.conf fil...

  • 0 kudos
RamlaS
by New Contributor II
  • 773 Views
  • 1 replies
  • 0 kudos

Resolved! Unity Catalog Pandas on Spark Limitation

According to Databricks UC Documentation, below are the some of the limitations on Shared Mode Cluster.1. In Databricks Runtime 13.3 LTS and above, Python scalar UDFs and Pandas UDFs are supported. Other Python UDFs, including UDAFs, UDTFs, and Panda...

  • 773 Views
  • 1 replies
  • 0 kudos
Latest Reply
raphaelblg
Honored Contributor II
  • 0 kudos

@RamlaS, UDFs on Unity Catalog is a feature that, at the current moment is still on the Public Preview stage. This means that the development has yet not finished.  UDFs can be used on DBR 13.3 and above, UDAFs are already available for DBR 15.2 on G...

  • 0 kudos
billraper
by New Contributor II
  • 596 Views
  • 2 replies
  • 0 kudos

New Users don't receive onboarding email

When I create a new user in Databricks, the new user does not receive their onboarding email.  It is not in their junkmail, deleted items or in their inbox.However, when I reset that user's password, they do receive the password reset link, and are a...

  • 596 Views
  • 2 replies
  • 0 kudos
Latest Reply
Rishabh_Tiwari
Community Manager
  • 0 kudos

Hi @billraper , I'm sorry to hear about the trouble. Would you mind sharing more about whether this is happening with a community portal profile or with a product profile? Please share the link to the profile with which you're experiencing this issue...

  • 0 kudos
1 More Replies
JoyceZhang
by New Contributor II
  • 489 Views
  • 2 replies
  • 0 kudos

cannot login to account management

HiI am not able to login to account  management (https://accounts.cloud.databricks.com), It somehow reninforce SSO, cannot login with username and password.  

  • 489 Views
  • 2 replies
  • 0 kudos
Latest Reply
Kaniz_Fatma
Community Manager
  • 0 kudos

Hi @JoyceZhang, Thank you for contacting Databricks Community Discussion Forum.   Please note that for any issues related to the Databricks Community Edition product, you can find helpful resources here. If you encounter any difficulties beyond what'...

  • 0 kudos
1 More Replies

Connect with Databricks Users in Your Area

Join a Regional User Group to connect with local Databricks users. Events will be happening in your city, and you won’t want to miss the chance to attend and share knowledge.

If there isn’t a group near you, start one and help create a community that brings people together.

Request a New Group
Labels