cancel
Showing results for 
Search instead for 
Did you mean: 
Administration & Architecture
Explore discussions on Databricks administration, deployment strategies, and architectural best practices. Connect with administrators and architects to optimize your Databricks environment for performance, scalability, and security.
cancel
Showing results for 
Search instead for 
Did you mean: 

Forum Posts

Learnit
by New Contributor II
  • 2034 Views
  • 0 replies
  • 0 kudos

Delta Sharing resulting in Bad_Request

Hi All,Recipeint encountering an issue while trying to access my organizational data (providers data) in a Delta Sharing scenario(databricks to databricks), and I'm hoping to get some guidance on how to resolve it. Here is the error message recipient...

  • 2034 Views
  • 0 replies
  • 0 kudos
m997al
by Contributor III
  • 1763 Views
  • 0 replies
  • 0 kudos

Azure Databricks with standard private link (only one Databricks authentication workspace)?

We have successfully set up Azure Databricks with standard private link (front-end and back-end).The front-end uses the authentication workspace as prescribed in the documentation.Suppose we use a "custom DNS" for the configuration below.  Can we set...

m997al_0-1710778785048.png
  • 1763 Views
  • 0 replies
  • 0 kudos
ducng
by New Contributor II
  • 1863 Views
  • 1 replies
  • 0 kudos

Unity catalog metastore and databricks workspace provisioning

 Hi,I am using terraform to create a databricks metastore using this.Now the issue is that I want to use this metastore reference in another terraform project where I create the databricks workspaces but there is no data source for metastore. Can any...

Administration & Architecture
metastore
Terraform
  • 1863 Views
  • 1 replies
  • 0 kudos
Chaitanya07
by New Contributor
  • 3870 Views
  • 0 replies
  • 0 kudos

Databricks Rest APIs CORS Issue

Hello Team,We are currently integrating Databricks Rest APIs into our in-house application for managing access permissions. While testing with curl and Postman, we've successfully accessed certain APIs like listing cluster permission. However, we're ...

  • 3870 Views
  • 0 replies
  • 0 kudos
VJ3
by Contributor
  • 5597 Views
  • 0 replies
  • 0 kudos

Workflow (Job) Cluster Permission Management

Hello Team,I understand that as the Job Owner, they can grant additional permissions to other users to manage/run/view the job.If "Can Manage" permission is given to the other users, that user can Edit the job including the Run-As parameter to themse...

  • 5597 Views
  • 0 replies
  • 0 kudos
alm
by New Contributor III
  • 1738 Views
  • 0 replies
  • 0 kudos

Cluster with shared access mode cannot query metastore

Hi, I have created a new UC enabled metastore using an Azure storage account and container connected to a Databricks workspace using an access connector. At first glance everything seems to work. I encounter a problem, however, when I try to query UC...

  • 1738 Views
  • 0 replies
  • 0 kudos
unauthenticated
by New Contributor
  • 4081 Views
  • 1 replies
  • 0 kudos

PyPI cluster libraries failing to get installed

Hi all,In my cluster, some of the PyPI cluster libraries started failing to get installed. It is weird because some of them get installed and some of are constantly failing. In every failed one, the error message is the same (just a package name is d...

  • 4081 Views
  • 1 replies
  • 0 kudos
Latest Reply
jacovangelder
Honored Contributor
  • 0 kudos

I've had this issue myself. What ended up to be the problem is I had windows line endings in my .sh script. You need to convert them to Linux line endings. 

  • 0 kudos
Kroy
by Contributor
  • 5757 Views
  • 4 replies
  • 1 kudos

Resolved! not able to see access schema inside system catalog

Hi,I am account admin role but not able to see access schema , I want to query audit log table     

Kroy_0-1710254944077.png
  • 5757 Views
  • 4 replies
  • 1 kudos
Latest Reply
Kroy
Contributor
  • 1 kudos

Done, needed to enable the system schema -- Run this command in notebook- to get list of available schema  %shcurl -v -X GET -H "Authorization: Bearer <PAT Token>" "https://adb-<xxx>.azuredatabricks.net/api/2.0/unity-catalog/metastores/<metastore-id>...

  • 1 kudos
3 More Replies
samini
by New Contributor II
  • 3246 Views
  • 2 replies
  • 0 kudos

Deleting MLflow runs programmatically

I would like to programmatically delete some MLflow runs based on a given run id. I am interested in the best practices on how to do this in Databricks workspaces.I know that MLflow cli has gc command which seems quite useful since it also deletes as...

  • 3246 Views
  • 2 replies
  • 0 kudos
Latest Reply
mameli
New Contributor II
  • 0 kudos

Do we have any updates on this? i'm using databricks and I have the same MlflowException

  • 0 kudos
1 More Replies
RoyRoger711
by New Contributor II
  • 2881 Views
  • 0 replies
  • 0 kudos

User Provisioning ( SCIM for OKTA)

Hello Databricks I wanted to ask a couple questions regarding switching SSO from onelogin to OKTA and turning on user provisioning. We have a total of 4 workspaces ( 1 sandbox , 2 dev and 1 prod) within our account. We have unified login enabled for ...

  • 2881 Views
  • 0 replies
  • 0 kudos
Jinyoung
by New Contributor
  • 2872 Views
  • 1 replies
  • 1 kudos

SQL warehouse realtime monitoring

A few days ago, our SQL warehouse had been back pressure and I knew that issue a hour later.I really wanna know how to get status of databricks SQL warehouse such as using DataDog agent not using web UI.

  • 2872 Views
  • 1 replies
  • 1 kudos
Latest Reply
Yev
New Contributor II
  • 1 kudos

Hi There! Is there any way to integrate Datadog with Databricks SQL Warehouse? I only found documentation related to clusters that seem to be different than SQL Warehouse in Databricks. However, I tried to add a global init script for Datadog integra...

  • 1 kudos
Sikalokym
by New Contributor II
  • 2227 Views
  • 4 replies
  • 0 kudos

Databricks job with a type "Python wheel" does not work if "Package name" contains dash

HelloI created a databricks job with a type "Python wheel". In the "Package name" field I assigned a python package which contains a dash in its name (see attach). The run of the job failed saying that could not import python package due to dash in t...

test_job.PNG
  • 2227 Views
  • 4 replies
  • 0 kudos
Latest Reply
AndréSalvati
New Contributor III
  • 0 kudos

There you can see a complete template project with a python wheel task and modules. Please, follow the instructions for deployment.https://github.com/andre-salvati/databricks-template

  • 0 kudos
3 More Replies
byrdman
by New Contributor III
  • 1445 Views
  • 1 replies
  • 1 kudos

having Problems starting a cluster on azure

I cant get the compute Tab to open.. just spins.   I also cant start a cluster from a notebook.  anyone else seeing this 

  • 1445 Views
  • 1 replies
  • 1 kudos
Latest Reply
Ayushi_Suthar
Databricks Employee
  • 1 kudos

Hi @Byron , Good Day!  Can you please provide more details on what you are seeing or facing when you are accessing the compute section?  Also please check with your workspace admin whether you have the correct cluster permission or not. You can refer...

  • 1 kudos
sampo
by New Contributor II
  • 1600 Views
  • 1 replies
  • 0 kudos

Databricks workspace creation using Terraform and storage configuration IAM Role Arn

Hi,When creating a new AWS Databricks workspace in account console manually, it appears that IAM Role ARN is mandatory parameter when configuring a Storage ConfigurationHowever in the Terraform databricks_mws_storage_configurations resource there is ...

sampo_1-1709404789587.png sampo_3-1709405245369.png
  • 1600 Views
  • 1 replies
  • 0 kudos
Latest Reply
sampo
New Contributor II
  • 0 kudos

Investigated this further... Terraform code to create the workspace resource "databricks_mws_workspaces" "databricks_ws" { provider = databricks.mws account_id = var.databricks_account_id aws_region = var.region workspace_name = "databricks-w...

  • 0 kudos
Priyam1
by New Contributor III
  • 3787 Views
  • 1 replies
  • 0 kudos

Editing the SPN name in databricks

As a workspace administrator, I'm finding that I cannot change the Service Principal Name (SPN) immediately after registering it through its application ID in Databricks. It appears that Databricks has linked the SPN name with its application ID and ...

  • 3787 Views
  • 1 replies
  • 0 kudos
Latest Reply
Ayushi_Suthar
Databricks Employee
  • 0 kudos

Hi @Priyam1 , Hope you are doing well!  You can change the display name of the Service Principal using the account SCIM APIs : https://docs.databricks.com/api/azure/account/accountserviceprincipals/patch Please let me know if this helps and leave a l...

  • 0 kudos