cancel
Showing results for 
Search instead for 
Did you mean: 
Administration & Architecture
Explore discussions on Databricks administration, deployment strategies, and architectural best practices. Connect with administrators and architects to optimize your Databricks environment for performance, scalability, and security.
cancel
Showing results for 
Search instead for 
Did you mean: 
Data + AI Summit 2024 - Data Lakehouse Architecture


Forum Posts

JoyceZhang
by New Contributor II
  • 540 Views
  • 1 replies
  • 0 kudos

cannot login to account management

HiI am not able to login to account  management (https://accounts.cloud.databricks.com), It somehow reninforce SSO, cannot login with username and password.  

  • 540 Views
  • 1 replies
  • 0 kudos
Latest Reply
" src="" />
This widget could not be displayed.
This widget could not be displayed.
This widget could not be displayed.
  • 0 kudos

This widget could not be displayed.
HiI am not able to login to account  management (https://accounts.cloud.databricks.com), It somehow reninforce SSO, cannot login with username and password.  

This widget could not be displayed.
  • 0 kudos
This widget could not be displayed.
camilo_s
by Contributor
  • 1224 Views
  • 1 replies
  • 1 kudos

Coarse-grained access management for jobs

Are there any perspectives in Databricks' roadmap for enabling coarse-grained access management for jobs?Currently, access to jobs has to be managed on a job by job basis: https://docs.databricks.com/en/security/auth-authz/access-control/index.html#j...

  • 1224 Views
  • 1 replies
  • 1 kudos
Latest Reply
camilo_s
Contributor
  • 1 kudos

Hi @Retired_mod, thanks for your reply.A more mature access management concept in Databricks would be definitely terrific. I understand it's not entirely along the AI-lines that Databricks is pushing hard currently, but it would improve a lot the pla...

  • 1 kudos
Carsten03
by New Contributor III
  • 15467 Views
  • 8 replies
  • 3 kudos

Resolved! Run workflow using git integration with service principal

Hi,I want to run a dbt workflow task and would like to use the git integration for that. Using my personal user I am able to do so but I am running my workflows using a service principal.I added git credentials and the repository using terraform. I a...

  • 15467 Views
  • 8 replies
  • 3 kudos
Latest Reply
camilo_s
Contributor
  • 3 kudos

I created that link using the "Share" button in the post but it's broken, sorry Here's a working link to the discussion: https://community.databricks.com/t5/data-engineering/git-credentials-for-service-principals-running-jobs/td-p/73802

  • 3 kudos
7 More Replies
Avvar2022
by Contributor
  • 3611 Views
  • 3 replies
  • 3 kudos

Is there a setting which restricts users from Creating Job and Pipeline?

as far i know currently ((as of 03-25-2024) databricks don't any workspace admin settings option to restrict users from creating a workflow/job or delta pipelines. Here is the use case for it Example: you have 3 tier landscape Dev, Qa and Prod.It is ...

Administration & Architecture
administration
jobs
pipelines
  • 3611 Views
  • 3 replies
  • 3 kudos
Latest Reply
camilo_s
Contributor
  • 3 kudos

 I notice that a separate discussion overlaps with the OPs issue: https://community.databricks.com/t5/data-engineering/restricting-workflow-creation-and-implementing-approval/td-p/4336@Retired_mod do you have a mechanism for clustering discussions fo...

  • 3 kudos
2 More Replies
RamlaS
by New Contributor II
  • 529 Views
  • 0 replies
  • 0 kudos

Can you all share your experiences on rolling out new features in Workspaces managed by you as Admin

As Data Platform Admins, do you follow some standard process or self service approach towards rolling out new features in your workspaces? Is the process automated? How is the testing done? Please share your thoughts. Given UC is introducing new feat...

  • 529 Views
  • 0 replies
  • 0 kudos
RozaZaharieva
by New Contributor
  • 4005 Views
  • 2 replies
  • 2 kudos

Get Azure Databricks Account ID

Hi everyone,Is it possible with Terraform or Azure CLI or any other not manual method to get the value for Azure Databricks Account ID and not to use manual method as is described here - https://learn.microsoft.com/en-us/azure/databricks/administrati...

Administration & Architecture
azuredatabricks
iac
Terraform
  • 4005 Views
  • 2 replies
  • 2 kudos
Latest Reply
SHeisterkamp
New Contributor II
  • 2 kudos

I am also looking for a solution to this. The path suggested by @Retired_mod does not work! When I run the proposed az cli, this is what I get:```$> az databricks workspace show --resource-group $my_rg --name $my_ws --query 'id'"/subscriptions/64e..<...

  • 2 kudos
1 More Replies
adb-rm
by New Contributor II
  • 1557 Views
  • 4 replies
  • 0 kudos

Unity catalog and metadata to backup site

Hi ,I am looking for Databricks DR(Disaster recovery)  site creation. My  primary setup is in west us and my primary data bricks created in primary site with unity catalog enable. with same meta data i want bring up the Databricks in my secondary sid...

  • 1557 Views
  • 4 replies
  • 0 kudos
Latest Reply
Walter_C
Databricks Employee
  • 0 kudos

As of now UC does not have DR setting, this is currently in our prioritized roadmap but no ETA is currently available. Some customers host an external metastore and replicate it across regions to achieve this.

  • 0 kudos
3 More Replies
RamlaS
by New Contributor II
  • 448 Views
  • 0 replies
  • 0 kudos

How are teams platform admin teams managing "allow list" for libraries feature UC?

We have so many team using maven libraries and are in the process of UC migration. These maven coordinates need to be added to "allow list" before they can be used in clusters. What is the standard process followed by admin teams for this feature? Do...

  • 448 Views
  • 0 replies
  • 0 kudos
mh_db
by New Contributor III
  • 457 Views
  • 0 replies
  • 0 kudos

link databrick to AWS GOV cloud

Does anyone know how to link databricks to AWS GOV cloud? I subscribed to databricks and created databricks account from the link in Marketplace. When I go to databricks account console the URL doesn't take me to the URL reference in the documentatio...

  • 457 Views
  • 0 replies
  • 0 kudos
mh_db
by New Contributor III
  • 526 Views
  • 0 replies
  • 0 kudos

Databricks AWS ACLs requirement

I'm going through the documentation in databricks for using existing VPC requirements. For the section that covers ACLs, what's the need to have this inbound rule: " ALLOW ALL from Source 0.0.0.0/0. This rule must be prioritized."https://docs.databri...

Administration & Architecture
Architecture
AWS
VPC
  • 526 Views
  • 0 replies
  • 0 kudos
Etyr
by Contributor
  • 1541 Views
  • 3 replies
  • 0 kudos

Unity Catalog read issue

Hello,Our company is POCing the Unity Catalog with Azure as provider.We have 2 subscriptions that contains 1 databricks each and 1 ADLS GEN2 each.Initially we have the default `hive_metastore` connected to the ADLS GEN2. I've created a secret scope a...

Etyr_0-1717660060863.png Etyr_1-1717660177530.png Etyr_2-1717663031618.png
  • 1541 Views
  • 3 replies
  • 0 kudos
Latest Reply
Etyr
Contributor
  • 0 kudos

Hello,The UC catalog container is sitting in Sub B.Basically we have a Spoke and Hub configuration for each Subscriptions. Each Subscriptions can access to any ressource inside it own subscription with some PE. But to access other Subscription ressou...

  • 0 kudos
2 More Replies
aghiya
by New Contributor
  • 697 Views
  • 1 replies
  • 0 kudos

Using JARs from Google Cloud Artifact Registry in Databricks for Job Execution

We have our CI/CD pipelines set up in Google Cloud using Cloud Build, and we are publishing our artifacts to a private repository in Google Cloud's Artifact Registry. I want to use these JAR files to create and run jobs in Databricks.However, when I ...

  • 697 Views
  • 1 replies
  • 0 kudos
Latest Reply
brijeshgoud1
New Contributor II
  • 0 kudos

To integrate your CI/CD pipeline with Databricks, you have a couple of options:1. Using Artifact Registry as an intermediary: Currently, Databricks does not directly support Google Artifact Registry URLs. However, you can use an intermediate storage ...

  • 0 kudos
tonypiazza
by New Contributor II
  • 1950 Views
  • 1 replies
  • 1 kudos

Resolved! Enable git credential for entire Databricks account

Hello,I have a question regarding git authentication in the context of a Databricks job. I know that a GitHub Personal Access Token can be generated for a user using GitHub App authorization flow. My team is currently configuring Terraform templates ...

  • 1950 Views
  • 1 replies
  • 1 kudos
Latest Reply
Yeshwanth
Databricks Employee
  • 1 kudos

@tonypiazza good day! To authorize access to all private repositories within a GitHub organization at the account level, without configuring git credentials on a per-user basis, follow these steps: 1. Use a GitHub Machine User:- Create a GitHub machi...

  • 1 kudos
Conlyn
by New Contributor III
  • 2439 Views
  • 2 replies
  • 0 kudos

Resolved! Terraform databricks_grants errors on external_location

We are using terraform to setup Unity Catalog external locations and when using databricks_grants to set permissions on the external locations it throws the following error:  Error: cannot create grants: permissions for external_location-test_locatio...

  • 2439 Views
  • 2 replies
  • 0 kudos
Latest Reply
Conlyn
New Contributor III
  • 0 kudos

I figured out my issue... The principal name is case sensitive and if the input value doesn't match the case of the email address or Group Name in the workspace/account it throws that ambiguous error.  

  • 0 kudos
1 More Replies

Connect with Databricks Users in Your Area

Join a Regional User Group to connect with local Databricks users. Events will be happening in your city, and you won’t want to miss the chance to attend and share knowledge.

If there isn’t a group near you, start one and help create a community that brings people together.

Request a New Group
Labels