cancel
Showing results for 
Search instead for 
Did you mean: 
Administration & Architecture
Explore discussions on Databricks administration, deployment strategies, and architectural best practices. Connect with administrators and architects to optimize your Databricks environment for performance, scalability, and security.
cancel
Showing results for 
Search instead for 
Did you mean: 

Forum Posts

mchirouze
by New Contributor
  • 1298 Views
  • 1 replies
  • 0 kudos

Send formatted html email from email distribution address

Hi, I have created an email distribution list "#MyList@mycompany.com". In the RShiny world I was able to send emails by a) getting the IP of the server I was sending the emails from and b) whitelisting that IP address within my company's SMTP Relay r...

  • 1298 Views
  • 1 replies
  • 0 kudos
Latest Reply
Kaniz
Community Manager
  • 0 kudos

Hi @mchirouze, To set up email services in Databricks, you have a few options depending on your requirements. Let’s explore them: Workspace Email Settings: As a workspace admin user, you can configure when users receive emails for certain events ...

  • 0 kudos
Learnit
by New Contributor II
  • 953 Views
  • 1 replies
  • 0 kudos

Delta Sharing resulting in Bad_Request

Hi All,Recipeint encountering an issue while trying to access my organizational data (providers data) in a Delta Sharing scenario(databricks to databricks), and I'm hoping to get some guidance on how to resolve it. Here is the error message recipient...

  • 953 Views
  • 1 replies
  • 0 kudos
Latest Reply
Kaniz
Community Manager
  • 0 kudos

Hi @Learnit, To resolve this issue, consider running the OPTIMIZE command on the shared Delta table to reduce the number of active files. After optimization, you may encounter the RemoveFiles limit if the OPTIMIZE command removed more than 100K files

  • 0 kudos
m997al
by Contributor
  • 1069 Views
  • 1 replies
  • 0 kudos

Azure Databricks with standard private link (only one Databricks authentication workspace)?

We have successfully set up Azure Databricks with standard private link (front-end and back-end).The front-end uses the authentication workspace as prescribed in the documentation.Suppose we use a "custom DNS" for the configuration below.  Can we set...

m997al_0-1710778785048.png
  • 1069 Views
  • 1 replies
  • 0 kudos
Latest Reply
Kaniz
Community Manager
  • 0 kudos

Hi @m997al, Your understanding is correct: The Azure Databricks authentication workspace is unique per Azure region, and its uniqueness extends to each Azure-region-per DNS zone. If you have further questions or need additional clarification, feel fr...

  • 0 kudos
ducng
by New Contributor II
  • 782 Views
  • 2 replies
  • 0 kudos

Unity catalog metastore and databricks workspace provisioning

 Hi,I am using terraform to create a databricks metastore using this.Now the issue is that I want to use this metastore reference in another terraform project where I create the databricks workspaces but there is no data source for metastore. Can any...

Administration & Architecture
metastore
Terraform
  • 782 Views
  • 2 replies
  • 0 kudos
Latest Reply
Kaniz
Community Manager
  • 0 kudos

Hi @ducng, When working with Databricks metastores in Terraform, you can create a metastore using the databricks_metastore resource. This metastore acts as a top-level container for objects in Unity Catalog, including data assets like tables and view...

  • 0 kudos
1 More Replies
VJ3
by New Contributor III
  • 1768 Views
  • 1 replies
  • 1 kudos

Resolved! Workflow (Job) Cluster Permission Management

Hello Team,I understand that as the Job Owner, they can grant additional permissions to other users to manage/run/view the job.If "Can Manage" permission is given to the other users, that user can Edit the job including the Run-As parameter to themse...

  • 1768 Views
  • 1 replies
  • 1 kudos
Latest Reply
Kaniz
Community Manager
  • 1 kudos

Hi @VJ3, Let’s delve into the permissions and behaviour of jobs in Databricks when it comes to managing and running them. “Can Manage” Permission: When a secondary user is granted the “Can Manage” permission on a job owned by the primary user, th...

  • 1 kudos
Chaitanya07
by New Contributor
  • 1309 Views
  • 1 replies
  • 0 kudos

Databricks Rest APIs CORS Issue

Hello Team,We are currently integrating Databricks Rest APIs into our in-house application for managing access permissions. While testing with curl and Postman, we've successfully accessed certain APIs like listing cluster permission. However, we're ...

  • 1309 Views
  • 1 replies
  • 0 kudos
Latest Reply
Kaniz
Community Manager
  • 0 kudos

Hi @Chaitanya07, Dealing with CORS (Cross-Origin Resource Sharing) issues can be a bit tricky, but I’ll provide some guidance to help you resolve this issue when integrating Databricks REST APIs into your in-house application. Understanding CORS:...

  • 0 kudos
Maxi1693
by New Contributor II
  • 1040 Views
  • 1 replies
  • 0 kudos

Error running 80 task at same time in Job, how limit this?

Hi! I have a Job running to process multiple streaming tables. In the beginning, it was working fine, but now I have 80 tables running in this job, the problem is that all the runs are trying to run at the same time throwing an error. Is there a way ...

  • 1040 Views
  • 1 replies
  • 0 kudos
Latest Reply
Kaniz
Community Manager
  • 0 kudos

Hi @Maxi1693, It appears that you’re encountering issues with parallel execution of tasks in your Databricks job. Let’s address this by considering a few strategies: Concurrency Limit for Tasks: Databricks allows a maximum of 1000 concurrent tas...

  • 0 kudos
SirCrayon
by New Contributor
  • 1121 Views
  • 1 replies
  • 0 kudos

Do shared clusters have multiple drivers?

Hi,I know that with single clusters, theres a single driver node and one driver per cluster. With shared clusters, multiple jobs can run concurrently. Does this still run on a single driver container or multiple driver containers run per application?...

  • 1121 Views
  • 1 replies
  • 0 kudos
Latest Reply
Kaniz
Community Manager
  • 0 kudos

Hi @SirCrayon,  In the context of shared clusters, where multiple jobs can run concurrently, the behavior regarding driver containers differs from that of single clusters. Single Clusters: In a single cluster, there is indeed a single driver node...

  • 0 kudos
alm
by New Contributor III
  • 888 Views
  • 1 replies
  • 0 kudos

Cluster with shared access mode cannot query metastore

Hi, I have created a new UC enabled metastore using an Azure storage account and container connected to a Databricks workspace using an access connector. At first glance everything seems to work. I encounter a problem, however, when I try to query UC...

  • 888 Views
  • 1 replies
  • 0 kudos
Latest Reply
Kaniz
Community Manager
  • 0 kudos

Hi @alm, It appears that you’re encountering issues related to the Hive metastore client when querying UC (Unified Catalog) using a shared access mode cluster in your Databricks workspace. Let’s troubleshoot this step by step: Hive Metastore ...

  • 0 kudos
unauthenticated
by New Contributor
  • 1229 Views
  • 2 replies
  • 0 kudos

PyPI cluster libraries failing to get installed

Hi all,In my cluster, some of the PyPI cluster libraries started failing to get installed. It is weird because some of them get installed and some of are constantly failing. In every failed one, the error message is the same (just a package name is d...

  • 1229 Views
  • 2 replies
  • 0 kudos
Latest Reply
jacovangelder
Contributor II
  • 0 kudos

I've had this issue myself. What ended up to be the problem is I had windows line endings in my .sh script. You need to convert them to Linux line endings. 

  • 0 kudos
1 More Replies
JJ11
by New Contributor
  • 1030 Views
  • 1 replies
  • 0 kudos

Connect Azure DevOps pipeline to Private Link Databricks Workspace

Hello, i have been trying to get a pipeline in Azure DevOps to communicate with a Databricks Workspace that uses private link connection. I have tried setting up a service connection using a service principle that is also attached to the workspace, h...

  • 1030 Views
  • 1 replies
  • 0 kudos
Latest Reply
Kaniz
Community Manager
  • 0 kudos

Hi @JJ11 , Integrating Azure DevOps with an Azure Databricks Workspace that uses a private link connection can be a bit tricky, but I’ll guide you through the process. Here are some steps and suggestions to help you achieve this: Azure Private Li...

  • 0 kudos
Kroy
by Contributor
  • 5024 Views
  • 2 replies
  • 0 kudos

How to Know DBU consumption in azure databricks ?

In Azure portal - Billing we can get the COST but how to know How much DBU is consumed ?

  • 5024 Views
  • 2 replies
  • 0 kudos
Latest Reply
Kaniz
Community Manager
  • 0 kudos

Hi @Kroy, To determine the Databricks Units (DBU) consumption in Azure Databricks, you can follow these steps:   Understanding DBUs: DBUs represent a unit of processing capability in Azure Databricks.They are billed on a per-second usage basis.The ac...

  • 0 kudos
1 More Replies
test-fall
by New Contributor
  • 796 Views
  • 1 replies
  • 0 kudos

unable to publish power bi dashboard with Databricks as source

Able to connect using personal access token and view the data in Hive; but when I publish to dashboard to share-point it results in :- PERMISSION_DENIED: You do not have permissions to autostart 1127-******-*********

  • 796 Views
  • 1 replies
  • 0 kudos
Latest Reply
Kaniz
Community Manager
  • 0 kudos

Hi @test-fall, The PERMISSION_DENIED error typically occurs when your user account lacks the necessary permissions to perform a specific action. In your case, it seems related to publishing a dashboard to SharePoint. Let’s troubleshoot this issue:...

  • 0 kudos
Kroy
by Contributor
  • 2100 Views
  • 4 replies
  • 1 kudos

Resolved! not able to see access schema inside system catalog

Hi,I am account admin role but not able to see access schema , I want to query audit log table     

Kroy_0-1710254944077.png
  • 2100 Views
  • 4 replies
  • 1 kudos
Latest Reply
Kroy
Contributor
  • 1 kudos

Done, needed to enable the system schema -- Run this command in notebook- to get list of available schema  %shcurl -v -X GET -H "Authorization: Bearer <PAT Token>" "https://adb-<xxx>.azuredatabricks.net/api/2.0/unity-catalog/metastores/<metastore-id>...

  • 1 kudos
3 More Replies
samini
by New Contributor II
  • 1812 Views
  • 3 replies
  • 0 kudos

Deleting MLflow runs programmatically

I would like to programmatically delete some MLflow runs based on a given run id. I am interested in the best practices on how to do this in Databricks workspaces.I know that MLflow cli has gc command which seems quite useful since it also deletes as...

  • 1812 Views
  • 3 replies
  • 0 kudos
Latest Reply
mameli
New Contributor II
  • 0 kudos

Do we have any updates on this? i'm using databricks and I have the same MlflowException

  • 0 kudos
2 More Replies
Join 100K+ Data Experts: Register Now & Grow with Us!

Excited to expand your horizons with us? Click here to Register and begin your journey to success!

Already a member? Login and join your local regional user group! If there isn’t one near you, fill out this form and we’ll create one for you to join!

Labels