cancel
Showing results for 
Search instead for 
Did you mean: 
Administration & Architecture
Explore discussions on Databricks administration, deployment strategies, and architectural best practices. Connect with administrators and architects to optimize your Databricks environment for performance, scalability, and security.
cancel
Showing results for 
Search instead for 
Did you mean: 
Data + AI Summit 2024 - Data Lakehouse Architecture


Forum Posts

cyborg-de
by New Contributor
  • 156 Views
  • 0 replies
  • 0 kudos

DNS resolution across vnet

Hi, I have created a new databricks workspace in Azure with backend private link. Settings are Required NSG rules - No Azure Databricks RuleNSG rules for AAD and azfrontdoor were added as per documentation. Private endpoint with subresource  databric...

  • 156 Views
  • 0 replies
  • 0 kudos
johnb1
by Contributor
  • 909 Views
  • 8 replies
  • 3 kudos

Resolved! Deploy Workflow only to specific target (Databricks Asset Bundles)

I am using Databricks Asset Bundles to deploy Databricks workflows to all of my target environments (dev, staging, prod). However, I have one specific workflow that is supposed to be deployed only to the dev target environment.How can I implement tha...

  • 909 Views
  • 8 replies
  • 3 kudos
Latest Reply
johnb1
Contributor
  • 3 kudos

Thanks for getting back and clarifying @szymon_dybczak 

  • 3 kudos
7 More Replies
MosefJJunior3rd
by New Contributor
  • 244 Views
  • 1 replies
  • 0 kudos

How to fully enable row-level concurrency on Databricks 14.1

Hey guys,I hope whoever's reading this is doing well.We're trying to enable row-level concurrency on Databricks 14.1. However, the documentation seems a bit contradictory as to whether this is possible, and whether the whole capability of row-level c...

  • 244 Views
  • 1 replies
  • 0 kudos
Latest Reply
gchandra
Databricks Employee
  • 0 kudos

Yes, your understanding is correct. As best practice, use 14.3 LTS DBR, as LTS has longer support compared to 6 months.

  • 0 kudos
JohnHarmer
by New Contributor
  • 156 Views
  • 1 replies
  • 0 kudos

Unable to publish notebook

Hello, I receive an error each time I attempt to publish my notebook. the error message does not provide any additional information on why it is occurring. I need to publish this for a bootcamp assignment, so please let me know what I can do to resol...

  • 156 Views
  • 1 replies
  • 0 kudos
Latest Reply
gchandra
Databricks Employee
  • 0 kudos

Try it from a different browser, or try it in incognito mode.

  • 0 kudos
saravjeet
by New Contributor III
  • 241 Views
  • 2 replies
  • 0 kudos

Allow non-admin users to view the driver logs from a Unity Catalog-enabled pipeline

We are trying to enable the option to allow log reading to the non-admin users in  the databricks wokspace but we are not able to see the obvious way to check them. The documentation is not showing after enabling the below property where to check the...

  • 241 Views
  • 2 replies
  • 0 kudos
Latest Reply
gchandra
Databricks Employee
  • 0 kudos

Do they see these links in the pipeline?

  • 0 kudos
1 More Replies
alexacas
by New Contributor II
  • 973 Views
  • 2 replies
  • 0 kudos

Resolved! Help with Databricks SQL Queries

Hi everyone,I’m relatively new to Databricks and trying to optimize some SQL queries for better performance. I’ve noticed that certain queries take longer to run than expected. Does anyone have tips or best practices for writing efficient SQL in Data...

  • 973 Views
  • 2 replies
  • 0 kudos
Latest Reply
filipniziol
Contributor
  • 0 kudos

Hi @alexacas ,The best thing is to share the queries and table structures But my general approach is:1. Use partitioning/zordering, or if you can upgrade runtime to 15.4, use liquid clustering, that is the new optimization technique.2. Make sure you ...

  • 0 kudos
1 More Replies
jmsimonramos1
by New Contributor II
  • 539 Views
  • 3 replies
  • 0 kudos

Grant permissions to groups on catalogs linked to the same metastore

Hi everyone!I am configuring several projects using Databricks, and I have a question regarding permission management in Unity Catalog. Here's the situation: Currently, I have two different Databricks resources in an Azure account, each with its resp...

Administration & Architecture
catalog
permissions
Unity Catalog
  • 539 Views
  • 3 replies
  • 0 kudos
Latest Reply
CharlesWoo
New Contributor II
  • 0 kudos

You could bind the catalog to specific workspaces, making them accessbile only from workspaces they are bound to.https://docs.databricks.com/en/catalogs/binding.htmlIn your example:if `my_catalog_2` is bound to `my_workspace_2` a user in `my_workspac...

  • 0 kudos
2 More Replies
Saf4Databricks
by New Contributor III
  • 170 Views
  • 1 replies
  • 0 kudos

Databricks Free Trial - can I link it back from AWS to Azure?

I created a 14-day trial account on Databricks.com and linked it to my AWS. I have not used the trial yet and want to link it to Azure instead. Question: Can I do it, and if so, how?Remark: Please note that I have not used trial yet and the only rele...

  • 170 Views
  • 1 replies
  • 0 kudos
Latest Reply
gchandra
Databricks Employee
  • 0 kudos

The easy way will be to create a new account in portal.azure.com and launch Azure Databricks with Pay as you go.

  • 0 kudos
d_kailthya
by New Contributor
  • 217 Views
  • 0 replies
  • 0 kudos

Implementing Databricks Persona in

Hi all,I am looking to implement the "persona" based access control across multiple workspaces for multiple user groups in Azure Databricks workspaces. Specifically,- I have a "DEV" workspace where the developer groups (Data Engineers and ML Engineer...

  • 217 Views
  • 0 replies
  • 0 kudos
Junda
by New Contributor II
  • 161 Views
  • 0 replies
  • 0 kudos

How to install private repository as package dependency in Databricks Workflow

I am a member of the development team in our company and we use Databricks as sort of like ETL tool. We utilize git integration for our program and run Workflow daily basis. Recently, we created another company internal private git repository and wan...

  • 161 Views
  • 0 replies
  • 0 kudos
mkieszek
by New Contributor II
  • 443 Views
  • 4 replies
  • 1 kudos

set access control list for job cluster

Hello,currently, we are using an init script that calls the DBW API to add "can_attach_to" permissions for a specific group to initialize the job cluster. How can we set an access control list for job clusters? Is it possible to add it to a policy?

  • 443 Views
  • 4 replies
  • 1 kudos
Latest Reply
szymon_dybczak
Contributor III
  • 1 kudos

Hi @mkieszek ,You can use REST API to set access control list for job clusters. The endpoint is for doing that you can find below/api/2.0/permissions/clusters/{cluster_id}  (https://docs.databricks.com/api/workspace/clusters/setpermissions)And it's n...

  • 1 kudos
3 More Replies
funsjanssen
by New Contributor III
  • 450 Views
  • 0 replies
  • 5 kudos

Azure Databricks Multi Tenant Solution

Hello Everyone,For the past few months, we’ve been extensively exploring the use of Databricks as the core of our data warehousing product. We provide analytics dashboards to other organizations and are particularly interested in the Column-Level Sec...

  • 450 Views
  • 0 replies
  • 5 kudos
AlbertWang
by Contributor
  • 481 Views
  • 5 replies
  • 2 kudos

Resolved! Databricks bundle custom variables feature does not work for workspace host

Hi all,I am not sure whether this is a bug. I followed the document.My databricks.yml file: bundle: name: test variables: DATABRICKS_HOST: description: The host of the Databricks workspace. DATABRICKS_JOB_RUNNING_SERVICE_PRINCIPAL_CLIENT_I...

  • 481 Views
  • 5 replies
  • 2 kudos
Latest Reply
filipniziol
Contributor
  • 2 kudos

Hi @AlbertWang ,The documentation states you cannot map workspace host: 

  • 2 kudos
4 More Replies
AndySkinner
by New Contributor III
  • 141 Views
  • 0 replies
  • 1 kudos

Troubleshooting Cluster

 We had a failure on a previously running fact table load (our biggest one) and it looked like an executor was failing due to a timeout error. As a test we upped the cluster size and changed the spark.executor.heartbeatinterval to 300s and the spark....

  • 141 Views
  • 0 replies
  • 1 kudos

Connect with Databricks Users in Your Area

Join a Regional User Group to connect with local Databricks users. Events will be happening in your city, and you won’t want to miss the chance to attend and share knowledge.

If there isn’t a group near you, start one and help create a community that brings people together.

Request a New Group
Labels