cancel
Showing results for 
Search instead for 
Did you mean: 
Administration & Architecture
Explore discussions on Databricks administration, deployment strategies, and architectural best practices. Connect with administrators and architects to optimize your Databricks environment for performance, scalability, and security.
cancel
Showing results for 
Search instead for 
Did you mean: 
Data + AI Summit 2024 - Data Lakehouse Architecture


Forum Posts

KevinGagnon
by New Contributor
  • 671 Views
  • 0 replies
  • 0 kudos

Delta live table : run_as

Does Databricks have any plans to decouple the owner from the "run_as" identity in Delta Live Table like it can be done in jobs?The problem arise specially when using DABs. The service principal used to deploy DLTs shouldn't be the owner AND the runn...

  • 671 Views
  • 0 replies
  • 0 kudos
macmiller1
by New Contributor II
  • 3371 Views
  • 0 replies
  • 0 kudos

Pass secret in spark config when value is in form a.b.c={{secrets/scope/secret}}

I am configuring the Cluster for a spark-submit task and I am trying to specify `spark.executor.extraJavaOptions a.b.c={{secrets/scope/secret}}` but the literal {{secrets/scope/secret}} is being passed in rather than the secret value itself.I know th...

  • 3371 Views
  • 0 replies
  • 0 kudos
sandipkumar
by New Contributor II
  • 919 Views
  • 1 replies
  • 0 kudos

Updating python version from 3.8 to 3.12 for s3 ingestion

I went to AWS Cloudformation stack and edited the template from python 3.8 to 3.12 and updated. I did this for both the workspace stack and the s3 ingestion stack. Will it break anything? Do I need to make any changes in the python code in the templa...

  • 919 Views
  • 1 replies
  • 0 kudos
Latest Reply
sandipkumar
New Contributor II
  • 0 kudos

Hi @Retired_mod ,Thanks a lot! I will look at StackSets.As I mentioned the code is not written by me but Databricks. Why does Databricks not use a new python version in its default stacks? We are low on resources and heavily rely on the default Datab...

  • 0 kudos
dbrx_user
by New Contributor III
  • 2039 Views
  • 3 replies
  • 5 kudos

Resolved! Move to 100% Serverless

Hi all,A few questions about the upcoming transition to 100% serverless, if anyone has any info that would be great!When will the move to serverless occur? I understand from 1st July (today) but has anyone seen a roadmap?What will the move to serverl...

  • 2039 Views
  • 3 replies
  • 5 kudos
Latest Reply
m997al
Contributor III
  • 5 kudos

Hi, so our Databricks contact just assured us the following, after we asked about this issue:Databricks is officially (but won’t be GA in every region till end of July) 100% Serverless OPTIONAL.We understand many of our customers have begged for 100%...

  • 5 kudos
2 More Replies
RamlaSuhra
by New Contributor II
  • 1035 Views
  • 1 replies
  • 0 kudos

Resolved! Unity Catalog Pandas on Spark Limitation

According to Databricks UC Documentation, below are the some of the limitations on Shared Mode Cluster.1. In Databricks Runtime 13.3 LTS and above, Python scalar UDFs and Pandas UDFs are supported. Other Python UDFs, including UDAFs, UDTFs, and Panda...

  • 1035 Views
  • 1 replies
  • 0 kudos
Latest Reply
raphaelblg
Databricks Employee
  • 0 kudos

@RamlaSuhra, UDFs on Unity Catalog is a feature that, at the current moment is still on the Public Preview stage. This means that the development has yet not finished.  UDFs can be used on DBR 13.3 and above, UDAFs are already available for DBR 15.2 ...

  • 0 kudos
billraper
by New Contributor II
  • 733 Views
  • 2 replies
  • 0 kudos

New Users don't receive onboarding email

When I create a new user in Databricks, the new user does not receive their onboarding email.  It is not in their junkmail, deleted items or in their inbox.However, when I reset that user's password, they do receive the password reset link, and are a...

  • 733 Views
  • 2 replies
  • 0 kudos
Latest Reply
Rishabh_Tiwari
Databricks Employee
  • 0 kudos

Hi @billraper , I'm sorry to hear about the trouble. Would you mind sharing more about whether this is happening with a community portal profile or with a product profile? Please share the link to the profile with which you're experiencing this issue...

  • 0 kudos
1 More Replies
JoyceZhang
by New Contributor II
  • 583 Views
  • 1 replies
  • 0 kudos

cannot login to account management

HiI am not able to login to account  management (https://accounts.cloud.databricks.com), It somehow reninforce SSO, cannot login with username and password.  

  • 583 Views
  • 1 replies
  • 0 kudos
Latest Reply
" src="" />
This widget could not be displayed.
This widget could not be displayed.
This widget could not be displayed.
  • 0 kudos

This widget could not be displayed.
HiI am not able to login to account  management (https://accounts.cloud.databricks.com), It somehow reninforce SSO, cannot login with username and password.  

This widget could not be displayed.
  • 0 kudos
This widget could not be displayed.
camilo_s
by Contributor
  • 1272 Views
  • 1 replies
  • 1 kudos

Coarse-grained access management for jobs

Are there any perspectives in Databricks' roadmap for enabling coarse-grained access management for jobs?Currently, access to jobs has to be managed on a job by job basis: https://docs.databricks.com/en/security/auth-authz/access-control/index.html#j...

  • 1272 Views
  • 1 replies
  • 1 kudos
Latest Reply
camilo_s
Contributor
  • 1 kudos

Hi @Retired_mod, thanks for your reply.A more mature access management concept in Databricks would be definitely terrific. I understand it's not entirely along the AI-lines that Databricks is pushing hard currently, but it would improve a lot the pla...

  • 1 kudos
Carsten03
by New Contributor III
  • 16547 Views
  • 8 replies
  • 3 kudos

Resolved! Run workflow using git integration with service principal

Hi,I want to run a dbt workflow task and would like to use the git integration for that. Using my personal user I am able to do so but I am running my workflows using a service principal.I added git credentials and the repository using terraform. I a...

  • 16547 Views
  • 8 replies
  • 3 kudos
Latest Reply
camilo_s
Contributor
  • 3 kudos

I created that link using the "Share" button in the post but it's broken, sorry Here's a working link to the discussion: https://community.databricks.com/t5/data-engineering/git-credentials-for-service-principals-running-jobs/td-p/73802

  • 3 kudos
7 More Replies
Avvar2022
by Contributor
  • 3900 Views
  • 3 replies
  • 3 kudos

Is there a setting which restricts users from Creating Job and Pipeline?

as far i know currently ((as of 03-25-2024) databricks don't any workspace admin settings option to restrict users from creating a workflow/job or delta pipelines. Here is the use case for it Example: you have 3 tier landscape Dev, Qa and Prod.It is ...

Administration & Architecture
administration
jobs
pipelines
  • 3900 Views
  • 3 replies
  • 3 kudos
Latest Reply
camilo_s
Contributor
  • 3 kudos

 I notice that a separate discussion overlaps with the OPs issue: https://community.databricks.com/t5/data-engineering/restricting-workflow-creation-and-implementing-approval/td-p/4336@Retired_mod do you have a mechanism for clustering discussions fo...

  • 3 kudos
2 More Replies
RamlaSuhra
by New Contributor II
  • 567 Views
  • 0 replies
  • 0 kudos

Can you all share your experiences on rolling out new features in Workspaces managed by you as Admin

As Data Platform Admins, do you follow some standard process or self service approach towards rolling out new features in your workspaces? Is the process automated? How is the testing done? Please share your thoughts. Given UC is introducing new feat...

  • 567 Views
  • 0 replies
  • 0 kudos
RozaZaharieva
by New Contributor
  • 4389 Views
  • 2 replies
  • 2 kudos

Get Azure Databricks Account ID

Hi everyone,Is it possible with Terraform or Azure CLI or any other not manual method to get the value for Azure Databricks Account ID and not to use manual method as is described here - https://learn.microsoft.com/en-us/azure/databricks/administrati...

Administration & Architecture
azuredatabricks
iac
Terraform
  • 4389 Views
  • 2 replies
  • 2 kudos
Latest Reply
SHeisterkamp
New Contributor II
  • 2 kudos

I am also looking for a solution to this. The path suggested by @Retired_mod does not work! When I run the proposed az cli, this is what I get:```$> az databricks workspace show --resource-group $my_rg --name $my_ws --query 'id'"/subscriptions/64e..<...

  • 2 kudos
1 More Replies
adb-rm
by New Contributor II
  • 1773 Views
  • 4 replies
  • 0 kudos

Unity catalog and metadata to backup site

Hi ,I am looking for Databricks DR(Disaster recovery)  site creation. My  primary setup is in west us and my primary data bricks created in primary site with unity catalog enable. with same meta data i want bring up the Databricks in my secondary sid...

  • 1773 Views
  • 4 replies
  • 0 kudos
Latest Reply
Walter_C
Databricks Employee
  • 0 kudos

As of now UC does not have DR setting, this is currently in our prioritized roadmap but no ETA is currently available. Some customers host an external metastore and replicate it across regions to achieve this.

  • 0 kudos
3 More Replies
RamlaSuhra
by New Contributor II
  • 497 Views
  • 0 replies
  • 0 kudos

How are teams platform admin teams managing "allow list" for libraries feature UC?

We have so many team using maven libraries and are in the process of UC migration. These maven coordinates need to be added to "allow list" before they can be used in clusters. What is the standard process followed by admin teams for this feature? Do...

  • 497 Views
  • 0 replies
  • 0 kudos

Connect with Databricks Users in Your Area

Join a Regional User Group to connect with local Databricks users. Events will be happening in your city, and you won’t want to miss the chance to attend and share knowledge.

If there isn’t a group near you, start one and help create a community that brings people together.

Request a New Group
Labels