cancel
Turn on suggestions
Auto-suggest helps you quickly narrow down your search results by suggesting possible matches as you type.
Showing results for 
Search instead for 
Did you mean: 
Help
Administration & Architecture
Explore discussions on Databricks administration, deployment strategies, and architectural best practices. Connect with administrators and architects to optimize your Databricks environment for performance, scalability, and security.
cancel
Turn on suggestions
Auto-suggest helps you quickly narrow down your search results by suggesting possible matches as you type.
Showing results for 
Search instead for 
Did you mean: 

Forum Posts

Start a conversation
RoyRoger711
by • New Contributor II
  • 1677 Views
  • 1 replies
  • 0 kudos

Resolved! User Provisioning ( SCIM for OKTA)

Hello Databricks I wanted to ask a couple questions regarding switching SSO from onelogin to OKTA and turning on user provisioning. We have a total of 4 workspaces ( 1 sandbox , 2 dev and 1 prod) within our account. We have unified login enabled for ...

Administration & Architecture
Reply
  • 1677 Views
  • 1 replies
  • 0 kudos
View Replies
Latest Reply
Kaniz
Community Manager
  • 0 kudos

Hi @RoyRoger711, Let’s break down your questions regarding switching SSO from OneLogin to Okta and enabling user provisioning for Databricks workspaces. Switching SSO from OneLogin to Okta: When transitioning from OneLogin to Okta for Single Sign...

  • 0 kudos
Jinyoung
by • New Contributor
  • 1408 Views
  • 2 replies
  • 0 kudos

SQL warehouse realtime monitoring

A few days ago, our SQL warehouse had been back pressure and I knew that issue a hour later.I really wanna know how to get status of databricks SQL warehouse such as using DataDog agent not using web UI.

Administration & Architecture
Reply
  • 1408 Views
  • 2 replies
  • 0 kudos
View Replies
Latest Reply
Yev
New Contributor II
  • 0 kudos

Hi There! Is there any way to integrate Datadog with Databricks SQL Warehouse? I only found documentation related to clusters that seem to be different than SQL Warehouse in Databricks. However, I tried to add a global init script for Datadog integra...

  • 0 kudos
1 More Replies
Sikalokym
by • New Contributor II
  • 1177 Views
  • 4 replies
  • 0 kudos

Databricks job with a type "Python wheel" does not work if "Package name" contains dash

HelloI created a databricks job with a type "Python wheel". In the "Package name" field I assigned a python package which contains a dash in its name (see attach). The run of the job failed saying that could not import python package due to dash in t...

test_job.PNG
Administration & Architecture
Bug
Reply
  • 1177 Views
  • 4 replies
  • 0 kudos
View Replies
Latest Reply
AndréSalvati
New Contributor III
  • 0 kudos

There you can see a complete template project with a python wheel task and modules. Please, follow the instructions for deployment.https://github.com/andre-salvati/databricks-template

  • 0 kudos
3 More Replies
byrdman
by • New Contributor III
  • 906 Views
  • 1 replies
  • 1 kudos

having Problems starting a cluster on azure

I cant get the compute Tab to open.. just spins.   I also cant start a cluster from a notebook.  anyone else seeing this 

Administration & Architecture
Reply
  • 906 Views
  • 1 replies
  • 1 kudos
View Replies
Latest Reply
Ayushi_Suthar
Honored Contributor
  • 1 kudos

Hi @Byron , Good Day!  Can you please provide more details on what you are seeing or facing when you are accessing the compute section?  Also please check with your workspace admin whether you have the correct cluster permission or not. You can refer...

  • 1 kudos
sampo
by • New Contributor II
  • 885 Views
  • 1 replies
  • 0 kudos

Databricks workspace creation using Terraform and storage configuration IAM Role Arn

Hi,When creating a new AWS Databricks workspace in account console manually, it appears that IAM Role ARN is mandatory parameter when configuring a Storage ConfigurationHowever in the Terraform databricks_mws_storage_configurations resource there is ...

sampo_1-1709404789587.png sampo_3-1709405245369.png
Administration & Architecture
Reply
  • 885 Views
  • 1 replies
  • 0 kudos
View Replies
Latest Reply
sampo
New Contributor II
  • 0 kudos

Investigated this further... Terraform code to create the workspace resource "databricks_mws_workspaces" "databricks_ws" { provider = databricks.mws account_id = var.databricks_account_id aws_region = var.region workspace_name = "databricks-w...

  • 0 kudos
Priyam1
by • New Contributor III
  • 2124 Views
  • 1 replies
  • 0 kudos

Editing the SPN name in databricks

As a workspace administrator, I'm finding that I cannot change the Service Principal Name (SPN) immediately after registering it through its application ID in Databricks. It appears that Databricks has linked the SPN name with its application ID and ...

Administration & Architecture
Reply
  • 2124 Views
  • 1 replies
  • 0 kudos
View Replies
Latest Reply
Ayushi_Suthar
Honored Contributor
  • 0 kudos

Hi @Priyam1 , Hope you are doing well!  You can change the display name of the Service Principal using the account SCIM APIs : https://docs.databricks.com/api/azure/account/accountserviceprincipals/patch Please let me know if this helps and leave a l...

  • 0 kudos
eric-cordeiro
by • New Contributor II
  • 4495 Views
  • 3 replies
  • 1 kudos

Databricks AWS Secrets Manager access

I have a workspace deployed in AWS and need to read some secrets from AWS Secrets Manager in my notebook. I'm aware that there is no default process similar to Azure Key Vault, however I know that we can try to access it using boto3, but I'm stuck at...

Administration & Architecture
Reply
  • 4495 Views
  • 3 replies
  • 1 kudos
View Replies
Latest Reply
Kaniz
Community Manager
  • 1 kudos

Hi @eric-cordeiro,  IAM roles are used for authentication to access AWS Secrets Manager from a Databricks Notebook in AWS. - Create a Cross-Account IAM Role with permissions to access secrets in AWS Secrets Manager.- Create an access policy that gran...

  • 1 kudos
2 More Replies
NadithK
by • Contributor
  • 3758 Views
  • 4 replies
  • 3 kudos

Creating a private link for DBFS root storage

I am facing an issue with securing root DBFS storage access from Databricks.As I understand, Azure Databricks creates a default blob storage (a.k.a root storage) during the deployment process which is used for storing logs and telemetry. This storage...

Administration & Architecture
dbfs
privatelink
Reply
  • 3758 Views
  • 4 replies
  • 3 kudos
View Replies
Latest Reply
PratikK
New Contributor II
  • 3 kudos

Hi @NadithK ,You need to create the private endpoint in the resource group where the workspace is deployed and not in the workspace-managed resource group. The workspace-managed resource group has the deny assignment which will not allow to create a ...

  • 3 kudos
3 More Replies
PallaviVetal
by • New Contributor II
  • 2872 Views
  • 7 replies
  • 0 kudos

Disable harmful commands at databricks cluster/workspace level

Hi,I want to prevent user from executing harmful commands like dbutils.fs.rm at cluster level/workspace level.Can we do this in databricks? Are there any classes/APIs that we can override?

Administration & Architecture
Reply
  • 2872 Views
  • 7 replies
  • 0 kudos
View Replies
Latest Reply
Nikhil_2800
New Contributor II
  • 0 kudos

Hi ,I got the similar kind of problem too.I followed this put chmod -x /bin/rm in init.sh in workspace and add it at cluster your cluster level init script or global init script.this restricts the users to use rm command.Regards,Nikhil Srivastava@Pal...

  • 0 kudos
6 More Replies
Mumrel
by • Contributor
  • 1599 Views
  • 3 replies
  • 0 kudos

Service Principal can be deleted but permissions not managed

On Azure I added a service principal X to my databricks workspace. I therefore had the Service Prinicpal Manager role on that service principal X. I accidentally downgraded my rights to Service Principal User and now can's get my Managers role back. ...

Mumrel_0-1708705677371.png
Administration & Architecture
Reply
  • 1599 Views
  • 3 replies
  • 0 kudos
View Replies
Latest Reply
Mumrel
Contributor
  • 0 kudos

I believe so because the described effect was also observed in another databricks workspace, where I did NOT make the accidental change

  • 0 kudos
2 More Replies
fradetjulien
by • New Contributor
  • 1923 Views
  • 1 replies
  • 0 kudos

SSO with Azure Active Directory : Authentication failed

I have a Databricks account with the E2 version of the Databricks platform. The unified login is not enabled on the account because it was created before June 21, 2023.I configured SSO authentication for the account and for a given workspace (separat...

Administration & Architecture
active directory
sso
Reply
  • 1923 Views
  • 1 replies
  • 0 kudos
View Replies
Latest Reply
146404
New Contributor II
  • 0 kudos

where you able to resolve this? we too face same error

  • 0 kudos
100804
by • New Contributor II
  • 1411 Views
  • 2 replies
  • 1 kudos

Instance Profile Access Controls

I manage instance profiles assigned to specific user groups. For example, instance profile A provides access solely to group A. Currently, any user within group A has the ability to update the permissions of a cluster using instance profile A, which ...

Administration & Architecture
Reply
  • 1411 Views
  • 2 replies
  • 1 kudos
View Replies
Latest Reply
100804
New Contributor II
  • 1 kudos

 Hi @Kaniz,Thank you for your guidance. I am following the strategies outlined in steps 1 and 2, and I remain concerned about a specific scenario.Consider instance profile A, which is designed to grant access exclusively to group A. If user A, a memb...

  • 1 kudos
1 More Replies
ossinova
by • Contributor II
  • 1022 Views
  • 2 replies
  • 0 kudos

Defaulting or overriding the cluster policy list order

I have numerous cluster policies varying that varies in sizes (Job - xsmall, Job - small, Job - medium...). However, when I create a new job and create a new job cluster the default policy selected from the drop down menu is on the bigger size. Is th...

Job cluster.png
Administration & Architecture
Reply
  • 1022 Views
  • 2 replies
  • 0 kudos
View Replies
Latest Reply
-werners-
Esteemed Contributor III
  • 0 kudos

I checked on our sorting, and there does not seem to be any logic in it.They are not sorted by ID or description for sure.

  • 0 kudos
1 More Replies
rfreitas
by • New Contributor II
  • 3814 Views
  • 1 replies
  • 1 kudos

Notebook and folder owner

Hi allWe can use this API https://docs.databricks.com/api/workspace/dbsqlpermissions/transferownership to transfer the ownership of a Query.Is there anything similar for notebooks and folders?

Administration & Architecture
Reply
  • 3814 Views
  • 1 replies
  • 1 kudos
View Replies
Latest Reply
feiyun0112
Contributor III
  • 1 kudos

Workspace object permissions — Manage which users can read, run, edit, or manage directories, files, and notebooks.https://docs.databricks.com/api/workspace/workspace/setpermissions

  • 1 kudos
RaulPino
by • New Contributor III
  • 3482 Views
  • 4 replies
  • 2 kudos

Resolved! Networking reduction cost for NATGateway and Shared Catalog

Use case and context:We have a databricks workspace in a specific region, reading and writing files from/to the same region.We also read from a Shared Catalog in a different company, a data provider, which is pointing to multi-region s3 buckets.The r...

Administration & Architecture
natgateway
networking
S3
shared catalog
VPC
Reply
  • 3482 Views
  • 4 replies
  • 2 kudos
View Replies
Latest Reply
RaulPino
New Contributor III
  • 2 kudos

Thanks @Kaniz for all the suggestions.After some days of monitoring NAT cost, I realized that the implementation of the S3 Gateway Endpoint it was actually working, the problem was that I thought that this change would be reflected right away in term...

  • 2 kudos
3 More Replies
Join 100K+ Data Experts: Register Now & Grow with Us!

Excited to expand your horizons with us? Click here to Register and begin your journey to success!

Already a member? Login and join your local regional user group! If there isn’t one near you, fill out this form and we’ll create one for you to join!

Labels