cancel
Showing results for 
Search instead for 
Did you mean: 
Administration & Architecture
Explore discussions on Databricks administration, deployment strategies, and architectural best practices. Connect with administrators and architects to optimize your Databricks environment for performance, scalability, and security.
cancel
Showing results for 
Search instead for 
Did you mean: 

Forum Posts

KLin
by New Contributor III
  • 296 Views
  • 6 replies
  • 1 kudos

Resolved! Unable to Pinpoint where network traffic originates from in GCP

Hi everyone,I have a question regarding networking. A bit of background first: For security reasons, the current allow-policy from GCP to our on-prem-infrastructure is being replaced by a deny-policy for traffic originating from GCP. Therefore access...

  • 296 Views
  • 6 replies
  • 1 kudos
Latest Reply
Alberto_Umana
Databricks Employee
  • 1 kudos

Hi @KLin, happy to help! -  The reason why traffic originates from the pods subnet for clusters/SQL warehouses without the x-databricks-nextgen-cluster tag (still using GKE) and from the node subnet for clusters with the GCE tag is due to the underly...

  • 1 kudos
5 More Replies
VJ3
by Contributor
  • 1125 Views
  • 2 replies
  • 1 kudos

Security Consideration for OAUTH Secrets to use Service Principal to authenticate with Databricks

What are the security consideration we need to keep in mind when we want to us OAUTH Secrets to use a Service Principal to access Azure Databricks when Identity federation is disabled and workspace is not yet on boarded on to Unity Catalog? Can we co...

  • 1125 Views
  • 2 replies
  • 1 kudos
Latest Reply
Rob_Lemmens
New Contributor III
  • 1 kudos

Any updates on this?Also struggling with the OAuth security considerations. Specifically with updating the OAuth Secrets.Currently using a SP to access Databricks workspace for DevOps purposes through the Databricks CLI.I have the SP set up to renew ...

  • 1 kudos
1 More Replies
Th0r
by New Contributor II
  • 212 Views
  • 4 replies
  • 0 kudos

Database Error in model Couldn't initialize file system for path abfss://

Recently the following error ocurs when running DBT:Database Error in model un_unternehmen_sat (models/2_un/partner/sats/un_unternehmen_sat.sql)Couldn't initialize file system for path abfss://dp-ext-fab@stcssdpextfabprd.dfs.core.windows.net/__unitys...

  • 212 Views
  • 4 replies
  • 0 kudos
Latest Reply
filipniziol
Contributor III
  • 0 kudos

Hi @Th0r ,Here is the explanation:Shallow clones in Databricks rely on references to data files of the original table. If the original table is dropped, recreated, or altered in a way that changes its underlying files, the shallow clone’s references ...

  • 0 kudos
3 More Replies
Databricks24
by New Contributor
  • 2151 Views
  • 2 replies
  • 0 kudos

UserAgentEntry added to JDBC URL but not visible in Audit logs

Hi,As part of Databricks Best Practices, I have added 'UserAgentEntry' to JDBC URL that is being created when we are executing SQL statements through the JDBC driver.Sample url - jdbc:databricks://<host>:443;httpPath=<httpPath>; AuthMech=3;UID=token;...

  • 2151 Views
  • 2 replies
  • 0 kudos
Latest Reply
satishdatagaps
New Contributor II
  • 0 kudos

Sorry, I was mistaken. please ignore the previous response. The correct one isjdbc:databricks://<host>:443;httpPath=<httpPath>; AuthMech=3;UID=token;PWD=<token>;UserAgentEntry=<ApplicationName/Year>;

  • 0 kudos
1 More Replies
PabloCSD
by Valued Contributor
  • 193 Views
  • 2 replies
  • 1 kudos

Resolved! Exhausted Server when deploying a Databricks Assets Bundle (DAB)

Hello, I'm currently with a colleague inspecting the code and when trying to deploy the DAB it gets stuck: (.venv) my_user@my_pc my-dab-project % databricks bundle deploy -t=dev -p=my-dab-project-prod Building wheel... Uploading my-dab-project-...

  • 193 Views
  • 2 replies
  • 1 kudos
Latest Reply
TinSlim
New Contributor III
  • 1 kudos

You are using a venv, the venv has too many files and is not needed to be included, try adding this on your databricks.ymlsync: exclude: - "venv" Hope it helps

  • 1 kudos
1 More Replies
dhruv1
by New Contributor II
  • 216 Views
  • 3 replies
  • 0 kudos

Delete the AWS Databricks account

I have created the aws databricks account from aws market place , but and I have cancelled the subscription after 14 days free trail from the market place. But still i see the account. How will i delete these databricks account associated with my ema...

  • 216 Views
  • 3 replies
  • 0 kudos
Latest Reply
TakuyaOmi
Valued Contributor
  • 0 kudos

@dhruv1 As mentioned, it would be best to reach out to support for assistance.https://help.databricks.com/s/signuprequest 

  • 0 kudos
2 More Replies
ac0
by Contributor
  • 1068 Views
  • 3 replies
  • 1 kudos

Delta Live Table pipeline steps explanation

Does anyone have documentation on what is actually occurring in each of these steps?Creating update Waiting for resourcesInitializingSetting up tablesRendering graphFor example, what is the difference between initializing and setting up tables? I am ...

  • 1068 Views
  • 3 replies
  • 1 kudos
Latest Reply
Mounika_Tarigop
Databricks Employee
  • 1 kudos

Yes, loading data (full refresh/refresh) into all streaming tables and refreshing materialized views are part of the "Setting up table" step in a Delta Live Tables (DLT) pipeline when running in trigger mode.In triggered mode, materialized views are ...

  • 1 kudos
2 More Replies
Dex
by New Contributor
  • 187 Views
  • 1 replies
  • 0 kudos

How to use Manged Identitify within Databricks Azure to access Blob Container?

Hi,My organization has asked that all blob storage accounts be accessed via managed identity. Several data brick notebooks are affected, so I'm currently trying to see how to set up a managed identity.We've added the Databricks resource provider to t...

  • 187 Views
  • 1 replies
  • 0 kudos
Latest Reply
Walter_C
Databricks Employee
  • 0 kudos

Have you followed the instructions available in docs https://learn.microsoft.com/en-us/azure/databricks/data-governance/unity-catalog/azure-managed-identities 

  • 0 kudos
JSilverberg
by New Contributor
  • 322 Views
  • 1 replies
  • 0 kudos

Resolved! Azure SCIM Provisioning Failures due to Resource Exhaustion

I had to make a significant change to the group membership in an Entra SCIM Provisioned group to Databricks, and the connector removed ALL users from the group (and sync'd) and then when I fixed, by adding a replacement dynamic group (with about 30% ...

  • 322 Views
  • 1 replies
  • 0 kudos
Latest Reply
parthSundarka
Databricks Employee
  • 0 kudos

Hi @JSilverberg , This occurs when the number of requests goes beyond the rate limits mentioned here. https://learn.microsoft.com/en-us/azure/databricks/resources/limits#:~:text=Identity,No To prevent these issues in the future, it is best to do thes...

  • 0 kudos
RohitKulkarni
by Contributor II
  • 123 Views
  • 1 replies
  • 0 kudos

Data Bricks Architect ceritification

Hello Team,I am planning to pursue the Databricks Architect certification. Could you please let me know which certification I should opt for?If you have any study material or relevant links, kindly share them.Your support would be highly appreciated....

  • 123 Views
  • 1 replies
  • 0 kudos
Latest Reply
Rjdudley
Contributor II
  • 0 kudos

The Architect credential is an accreditation, not a certification.  Accreditations are less rigorous and less expensive than certifications.  You didn't say which platform you are on, so here are links to the learning plans (which have the exams) for...

  • 0 kudos
F_Goudarzi
by New Contributor III
  • 391 Views
  • 1 replies
  • 1 kudos

Pros and cons of putting all various Databricks workspaces (dev, qa , prod) under one metastore

Hi there, If we have separate workspaces for each and every environment, then how we should go about structuring the metastore? What are the pros and cons of putting all workspaces under one metastore instead of having separate metastore for each?Tha...

  • 391 Views
  • 1 replies
  • 1 kudos
Latest Reply
Walter_C
Databricks Employee
  • 1 kudos

Hello Fatima, many thanks for your question. Please first note that if all the workspaces belong to the same account id and are on the same cloud region, they will all need to be associated with the same metastore as you can only have 1 metastore per...

  • 1 kudos
Dnirmania
by New Contributor III
  • 742 Views
  • 7 replies
  • 5 kudos

Resolved! Databricks best practices for azure storage account

Hello EveryoneCurrently, We are in process of building azure databricks and have some doubt regarding best practices to follow for azure storage account which we will be using to store data. Can anyone help me finding best practices to follow for sto...

  • 742 Views
  • 7 replies
  • 5 kudos
Latest Reply
bhanu_gautam
Contributor
  • 5 kudos

Thanks for sharing @Rjdudley @szymon_dybczak @filipniziol 

  • 5 kudos
6 More Replies
enr0c
by New Contributor
  • 219 Views
  • 1 replies
  • 0 kudos

Budget Policy - Service Principals don't seem to be allowed to use budget policies

ObjectiveTransfer existing DLT pipeline to new owner (service principal). Budget policies enabled.Steps to reproduceCreated a service principalAssigned it group membership of a group that is allowed to use a budget policyEnsured it has access to the ...

a203a1bb-80d2-4c51-bc2f-dbd379b6e5e8.png Screenshot 2024-11-20 090829.png Screenshot 2024-11-20 090938.png Screenshot 2024-11-20 091107.png
Administration & Architecture
budget-policy
service-principal
  • 219 Views
  • 1 replies
  • 0 kudos
Latest Reply
Abishek
Databricks Employee
  • 0 kudos

“Thanks for your question! I’m looking into this and will get back to you.”

  • 0 kudos
chethankumar
by New Contributor III
  • 348 Views
  • 3 replies
  • 1 kudos

Resolved! How to add existing recipient to existing delta share

I created a recipient in the Databricks console and also set up a Delta Share. Now, I’d like to link this existing recipient to the Delta Share. Is there a way to accomplish this using Terraform?

  • 348 Views
  • 3 replies
  • 1 kudos
Latest Reply
chethankumar
New Contributor III
  • 1 kudos

Hi @Naïm Thanks for your response. It seems your answer is helping me, but I'm facing another issue. The owner of my recipient is a group, not an individual user. I'm running this Terraform script using a service principal that is a member of that gr...

  • 1 kudos
2 More Replies

Connect with Databricks Users in Your Area

Join a Regional User Group to connect with local Databricks users. Events will be happening in your city, and you won’t want to miss the chance to attend and share knowledge.

If there isn’t a group near you, start one and help create a community that brings people together.

Request a New Group
Labels