cancel
Showing results for 
Search instead for 
Did you mean: 
Administration & Architecture
Explore discussions on Databricks administration, deployment strategies, and architectural best practices. Connect with administrators and architects to optimize your Databricks environment for performance, scalability, and security.
cancel
Showing results for 
Search instead for 
Did you mean: 
Data + AI Summit 2024 - Data Lakehouse Architecture


Forum Posts

alm
by New Contributor III
  • 1083 Views
  • 1 replies
  • 0 kudos

Cluster with shared access mode cannot query metastore

Hi, I have created a new UC enabled metastore using an Azure storage account and container connected to a Databricks workspace using an access connector. At first glance everything seems to work. I encounter a problem, however, when I try to query UC...

  • 1083 Views
  • 1 replies
  • 0 kudos
Latest Reply
Kaniz_Fatma
Community Manager
  • 0 kudos

Hi @alm, It appears that you’re encountering issues related to the Hive metastore client when querying UC (Unified Catalog) using a shared access mode cluster in your Databricks workspace. Let’s troubleshoot this step by step: Hive Metastore ...

  • 0 kudos
unauthenticated
by New Contributor
  • 1771 Views
  • 2 replies
  • 0 kudos

PyPI cluster libraries failing to get installed

Hi all,In my cluster, some of the PyPI cluster libraries started failing to get installed. It is weird because some of them get installed and some of are constantly failing. In every failed one, the error message is the same (just a package name is d...

  • 1771 Views
  • 2 replies
  • 0 kudos
Latest Reply
jacovangelder
Contributor III
  • 0 kudos

I've had this issue myself. What ended up to be the problem is I had windows line endings in my .sh script. You need to convert them to Linux line endings. 

  • 0 kudos
1 More Replies
JJ11
by New Contributor
  • 1409 Views
  • 1 replies
  • 0 kudos

Connect Azure DevOps pipeline to Private Link Databricks Workspace

Hello, i have been trying to get a pipeline in Azure DevOps to communicate with a Databricks Workspace that uses private link connection. I have tried setting up a service connection using a service principle that is also attached to the workspace, h...

  • 1409 Views
  • 1 replies
  • 0 kudos
Latest Reply
Kaniz_Fatma
Community Manager
  • 0 kudos

Hi @JJ11 , Integrating Azure DevOps with an Azure Databricks Workspace that uses a private link connection can be a bit tricky, but I’ll guide you through the process. Here are some steps and suggestions to help you achieve this: Azure Private Li...

  • 0 kudos
Kroy
by Contributor
  • 10901 Views
  • 2 replies
  • 0 kudos

How to Know DBU consumption in azure databricks ?

In Azure portal - Billing we can get the COST but how to know How much DBU is consumed ?

  • 10901 Views
  • 2 replies
  • 0 kudos
Latest Reply
Kaniz_Fatma
Community Manager
  • 0 kudos

Hi @Kroy, To determine the Databricks Units (DBU) consumption in Azure Databricks, you can follow these steps:   Understanding DBUs: DBUs represent a unit of processing capability in Azure Databricks.They are billed on a per-second usage basis.The ac...

  • 0 kudos
1 More Replies
test-fall
by New Contributor
  • 988 Views
  • 1 replies
  • 0 kudos

unable to publish power bi dashboard with Databricks as source

Able to connect using personal access token and view the data in Hive; but when I publish to dashboard to share-point it results in :- PERMISSION_DENIED: You do not have permissions to autostart 1127-******-*********

  • 988 Views
  • 1 replies
  • 0 kudos
Latest Reply
Kaniz_Fatma
Community Manager
  • 0 kudos

Hi @test-fall, The PERMISSION_DENIED error typically occurs when your user account lacks the necessary permissions to perform a specific action. In your case, it seems related to publishing a dashboard to SharePoint. Let’s troubleshoot this issue:...

  • 0 kudos
Kroy
by Contributor
  • 2668 Views
  • 4 replies
  • 1 kudos

Resolved! not able to see access schema inside system catalog

Hi,I am account admin role but not able to see access schema , I want to query audit log table     

Kroy_0-1710254944077.png
  • 2668 Views
  • 4 replies
  • 1 kudos
Latest Reply
Kroy
Contributor
  • 1 kudos

Done, needed to enable the system schema -- Run this command in notebook- to get list of available schema  %shcurl -v -X GET -H "Authorization: Bearer <PAT Token>" "https://adb-<xxx>.azuredatabricks.net/api/2.0/unity-catalog/metastores/<metastore-id>...

  • 1 kudos
3 More Replies
samini
by New Contributor II
  • 2209 Views
  • 3 replies
  • 0 kudos

Deleting MLflow runs programmatically

I would like to programmatically delete some MLflow runs based on a given run id. I am interested in the best practices on how to do this in Databricks workspaces.I know that MLflow cli has gc command which seems quite useful since it also deletes as...

  • 2209 Views
  • 3 replies
  • 0 kudos
Latest Reply
mameli
New Contributor II
  • 0 kudos

Do we have any updates on this? i'm using databricks and I have the same MlflowException

  • 0 kudos
2 More Replies
RoyRoger711
by New Contributor II
  • 1952 Views
  • 1 replies
  • 0 kudos

Resolved! User Provisioning ( SCIM for OKTA)

Hello Databricks I wanted to ask a couple questions regarding switching SSO from onelogin to OKTA and turning on user provisioning. We have a total of 4 workspaces ( 1 sandbox , 2 dev and 1 prod) within our account. We have unified login enabled for ...

  • 1952 Views
  • 1 replies
  • 0 kudos
Latest Reply
Kaniz_Fatma
Community Manager
  • 0 kudos

Hi @RoyRoger711, Let’s break down your questions regarding switching SSO from OneLogin to Okta and enabling user provisioning for Databricks workspaces. Switching SSO from OneLogin to Okta: When transitioning from OneLogin to Okta for Single Sign...

  • 0 kudos
Jinyoung
by New Contributor
  • 1734 Views
  • 2 replies
  • 0 kudos

SQL warehouse realtime monitoring

A few days ago, our SQL warehouse had been back pressure and I knew that issue a hour later.I really wanna know how to get status of databricks SQL warehouse such as using DataDog agent not using web UI.

  • 1734 Views
  • 2 replies
  • 0 kudos
Latest Reply
Yev
New Contributor II
  • 0 kudos

Hi There! Is there any way to integrate Datadog with Databricks SQL Warehouse? I only found documentation related to clusters that seem to be different than SQL Warehouse in Databricks. However, I tried to add a global init script for Datadog integra...

  • 0 kudos
1 More Replies
Sikalokym
by New Contributor II
  • 1363 Views
  • 4 replies
  • 0 kudos

Databricks job with a type "Python wheel" does not work if "Package name" contains dash

HelloI created a databricks job with a type "Python wheel". In the "Package name" field I assigned a python package which contains a dash in its name (see attach). The run of the job failed saying that could not import python package due to dash in t...

test_job.PNG
  • 1363 Views
  • 4 replies
  • 0 kudos
Latest Reply
AndréSalvati
New Contributor III
  • 0 kudos

There you can see a complete template project with a python wheel task and modules. Please, follow the instructions for deployment.https://github.com/andre-salvati/databricks-template

  • 0 kudos
3 More Replies
byrdman
by New Contributor III
  • 994 Views
  • 1 replies
  • 1 kudos

having Problems starting a cluster on azure

I cant get the compute Tab to open.. just spins.   I also cant start a cluster from a notebook.  anyone else seeing this 

  • 994 Views
  • 1 replies
  • 1 kudos
Latest Reply
Ayushi_Suthar
Honored Contributor
  • 1 kudos

Hi @Byron , Good Day!  Can you please provide more details on what you are seeing or facing when you are accessing the compute section?  Also please check with your workspace admin whether you have the correct cluster permission or not. You can refer...

  • 1 kudos
sampo
by New Contributor II
  • 989 Views
  • 1 replies
  • 0 kudos

Databricks workspace creation using Terraform and storage configuration IAM Role Arn

Hi,When creating a new AWS Databricks workspace in account console manually, it appears that IAM Role ARN is mandatory parameter when configuring a Storage ConfigurationHowever in the Terraform databricks_mws_storage_configurations resource there is ...

sampo_1-1709404789587.png sampo_3-1709405245369.png
  • 989 Views
  • 1 replies
  • 0 kudos
Latest Reply
sampo
New Contributor II
  • 0 kudos

Investigated this further... Terraform code to create the workspace resource "databricks_mws_workspaces" "databricks_ws" { provider = databricks.mws account_id = var.databricks_account_id aws_region = var.region workspace_name = "databricks-w...

  • 0 kudos
Priyam1
by New Contributor III
  • 2274 Views
  • 1 replies
  • 0 kudos

Editing the SPN name in databricks

As a workspace administrator, I'm finding that I cannot change the Service Principal Name (SPN) immediately after registering it through its application ID in Databricks. It appears that Databricks has linked the SPN name with its application ID and ...

  • 2274 Views
  • 1 replies
  • 0 kudos
Latest Reply
Ayushi_Suthar
Honored Contributor
  • 0 kudos

Hi @Priyam1 , Hope you are doing well!  You can change the display name of the Service Principal using the account SCIM APIs : https://docs.databricks.com/api/azure/account/accountserviceprincipals/patch Please let me know if this helps and leave a l...

  • 0 kudos
eric-cordeiro
by New Contributor II
  • 5119 Views
  • 3 replies
  • 1 kudos

Databricks AWS Secrets Manager access

I have a workspace deployed in AWS and need to read some secrets from AWS Secrets Manager in my notebook. I'm aware that there is no default process similar to Azure Key Vault, however I know that we can try to access it using boto3, but I'm stuck at...

  • 5119 Views
  • 3 replies
  • 1 kudos
Latest Reply
Kaniz_Fatma
Community Manager
  • 1 kudos

Hi @eric-cordeiro,  IAM roles are used for authentication to access AWS Secrets Manager from a Databricks Notebook in AWS. - Create a Cross-Account IAM Role with permissions to access secrets in AWS Secrets Manager.- Create an access policy that gran...

  • 1 kudos
2 More Replies
NadithK
by Contributor
  • 4173 Views
  • 4 replies
  • 3 kudos

Creating a private link for DBFS root storage

I am facing an issue with securing root DBFS storage access from Databricks.As I understand, Azure Databricks creates a default blob storage (a.k.a root storage) during the deployment process which is used for storing logs and telemetry. This storage...

  • 4173 Views
  • 4 replies
  • 3 kudos
Latest Reply
PratikK
New Contributor III
  • 3 kudos

Hi @NadithK ,You need to create the private endpoint in the resource group where the workspace is deployed and not in the workspace-managed resource group. The workspace-managed resource group has the deny assignment which will not allow to create a ...

  • 3 kudos
3 More Replies
Join 100K+ Data Experts: Register Now & Grow with Us!

Excited to expand your horizons with us? Click here to Register and begin your journey to success!

Already a member? Login and join your local regional user group! If there isn’t one near you, fill out this form and we’ll create one for you to join!

Labels