cancel
Turn on suggestions
Auto-suggest helps you quickly narrow down your search results by suggesting possible matches as you type.
Showing results for 
Search instead for 
Did you mean: 
Help
Administration & Architecture
Explore discussions on Databricks administration, deployment strategies, and architectural best practices. Connect with administrators and architects to optimize your Databricks environment for performance, scalability, and security.
cancel
Turn on suggestions
Auto-suggest helps you quickly narrow down your search results by suggesting possible matches as you type.
Showing results for 
Search instead for 
Did you mean: 

Forum Posts

Start a conversation
anurag_dev
by • New Contributor
  • 798 Views
  • 1 replies
  • 0 kudos

getting alerts in AWS UnauthorizedAccess:IAMUser/InstanceCredentialExfiltration.InsideAWS

Hi folks, I have been using IAM instance profile as a credentials provider to Databricks workspace. I am getting this error "Credentials created exclusively for an EC2 instance using instance role <role-arn> have been used from a remote AWS account 4...

Administration & Architecture
AWS
Reply
  • 798 Views
  • 1 replies
  • 0 kudos
View Replies
Latest Reply
Yeshwanth
Honored Contributor
  • 0 kudos

Hi @anurag_dev  Databricks runs a commit service that coordinates writes to Amazon S3 from multiple clusters. This service runs in the Databricks control plane. (441371677306)The S3 commit service helps guarantee consistency of writes across multiple...

  • 0 kudos
bas
by • New Contributor III
  • 2958 Views
  • 9 replies
  • 2 kudos

Resolved! Adding a service principal to workspace fails

Trying to add an Azure Service Principal to a workspace fails, whether via the web UI or the databricks cli. With the databricks, the console gives this:databricks service-principals -p ************* create --application-id ********-****-****-****-**...

Administration & Architecture
azure
Security
Reply
  • 2958 Views
  • 9 replies
  • 2 kudos
View Replies
Latest Reply
brickmaker
New Contributor II
  • 2 kudos

Same issue in AWS Databricks.

  • 2 kudos
8 More Replies
msserpa
by • New Contributor II
  • 2206 Views
  • 2 replies
  • 0 kudos

Can we turn off Playground & Marketplace for some users?

Hi Everyone,Hope all is well with you!I'm reaching out to the community for some advice on customizing our workspace settings. Specifically, I have two things I'm trying to figure out:Disabling the "Playground" Option: Is there a way to turn off the ...

169300_1-1704556176241.png
Administration & Architecture
Databricks
Marketplace
Playground
Reply
  • 2206 Views
  • 2 replies
  • 0 kudos
View Replies
Latest Reply
Wing139
New Contributor II
  • 0 kudos

how can we turn on the playground option, i cannot find it

  • 0 kudos
1 More Replies
sl79
by • New Contributor II
  • 1589 Views
  • 1 replies
  • 1 kudos

Unable to start a SQL cluster

We have a GCP databricks hosted deployment. last night our SQL warehouse failed to start and now just sits with this message:"Cluster Start-up Delayed. Please wait while we continue to try and start the cluster. No action is required from you"It didn...

Administration & Architecture
Reply
  • 1589 Views
  • 1 replies
  • 1 kudos
View Replies
Latest Reply
jose_gonzalez
Moderator
  • 1 kudos

Could you share the driver logs if you have them??

  • 1 kudos
Learnit
by • New Contributor II
  • 1496 Views
  • 1 replies
  • 0 kudos

Managing databricks workspace permissions

I need assistance with writing API/Python code to manage a Databricks workspace permissions database(unity catalog). The task involves obtaining a list of workspace details from the account console, which includes various details like Workspace name,...

Administration & Architecture
Reply
  • 1496 Views
  • 1 replies
  • 0 kudos
View Replies
Latest Reply
Kaniz_Fatma
Community Manager
  • 0 kudos

Hi @Learnit, Here’s a high-level overview of the steps you might take:   Obtain Workspace Details: You can use the Databricks REST API to obtain workspace details. The API provides various endpoints to manage access for different objects and users. M...

  • 0 kudos
pawelzak
by • New Contributor III
  • 809 Views
  • 2 replies
  • 0 kudos

Dashboard programmatical update

Hi,The question that I would like to answer is: How can one replicate dashboards across different environments? Ideally I would like to be able to store the definition of a dashboard in my repository and upload it to Databricks with code: this could ...

Administration & Architecture
Reply
  • 809 Views
  • 2 replies
  • 0 kudos
View Replies
Latest Reply
shan_chandra
Esteemed Contributor
  • 0 kudos

@pawelzak  - The Dashboards can not be exported using workspace /export API. 

  • 0 kudos
1 More Replies
m997al
by • Contributor
  • 2568 Views
  • 4 replies
  • 0 kudos

Azure Databricks with metastore, cannot create managed table

We have set up Azure Databricks with Unity Catalog (metastore) in an ADLS Gen2 storage account.Used Managed Identity (Databricks Access Connector) for connection from workspace(s) to ADLS Gen2ADLS Gen2 storage account has Storage Blob Data Owner and ...

m997al_0-1697803440437.png
Administration & Architecture
Reply
  • 2568 Views
  • 4 replies
  • 0 kudos
View Replies
Latest Reply
Kris2
New Contributor II
  • 0 kudos

@m997al Were you able to resolve this issue? I have same issue as you described. I am able to upload into Volumes but not able to create managed tables in ADLS metastore storage account..

  • 0 kudos
3 More Replies
Matt4
by • New Contributor III
  • 2470 Views
  • 2 replies
  • 0 kudos

Resolved! System tables in Azure with private link

Hi,Is there any restriction on what is able to be enabled with System tables when using Azure Databricks with a private link connection?  And is there a way around the restriction (that is secure)?In a public network I am able to enable the following...

Administration & Architecture
Reply
  • 2470 Views
  • 2 replies
  • 0 kudos
View Replies
Latest Reply
Matt4
New Contributor III
  • 0 kudos

Turns out it was an issue with Microsoft somewhere, they've now fixed it and all the system schemas have been enabled that I can enable"schemas":[{"schema":"operational_data","state":"UNAVAILABLE"},{"schema":"access","state":"ENABLE_COMPLETED"},{"sch...

  • 0 kudos
1 More Replies
kartikaybagla
by • New Contributor II
  • 3836 Views
  • 5 replies
  • 2 kudos

Unable to access all schemas in system catalog

https://docs.databricks.com/en/administration-guide/system-tables/index.htmlBased on this documentation, there should be a bunch of schemas in the system catalog and when trying to list them all, I can only see the INFORMATION_SCHEMA in the system ca...

Administration & Architecture
Reply
  • 3836 Views
  • 5 replies
  • 2 kudos
View Replies
Latest Reply
SanketMehta
New Contributor II
  • 2 kudos

Hi Team, If you have any particular solution for this, then please share as I am also unable to access any system tables I enabled in 2023 August now!!

  • 2 kudos
4 More Replies
pawelzak
by • New Contributor III
  • 707 Views
  • 2 replies
  • 0 kudos

Dashboard update through API

Hi,I would like to create / update dashboard definition based on the json file. How can one do it? I tried the following:databricks api post /api/2.0/preview/sql/dashboards/$dashboard_id --json @file.jsonBut it does not update the widgetsdatabricks a...

Administration & Architecture
Reply
  • 707 Views
  • 2 replies
  • 0 kudos
View Replies
Latest Reply
Kaniz_Fatma
Community Manager
  • 0 kudos

Hi @pawelzak, Databricks does not currently support creating or updating dashboards directly from a JSON file via their API.   For creating or updating resources in Databricks, you might want to look into the Databricks REST API. For instance, the Wo...

  • 0 kudos
1 More Replies
ilarsen
by • Contributor
  • 1248 Views
  • 3 replies
  • 0 kudos

When to add Users Groups or SPs from Account to Workspace

Hi community We are using Unity Catalog, SCIM and Identity Federation, so we have users, groups and service principals at Account level.  In what scenarios do users, groups and service principals need explicitly added to a Workspace?

Administration & Architecture
Reply
  • 1248 Views
  • 3 replies
  • 0 kudos
View Replies
Latest Reply
adamjosh
New Contributor II
  • 0 kudos

In the context of Unity Catalog, SCIM (System for Cross-domain Identity Management), and Identity Federation in a Unity-based environment, users, groups, and service principals may need to be explicitly added to a Workspace in various scenarios based...

  • 0 kudos
2 More Replies
bric-a-brac
by • New Contributor
  • 1805 Views
  • 1 replies
  • 0 kudos

Approval required when connecting Tableau Desktop to Azure Databricks

My users want to connect Tableau Desktop to Azure Databricks. I have tested this myself as Administrator of the databricks workspace and it works fine. One of my users can connect to Azure databricks fine and run notebooks. However when he connects T...

image-20240105-123145.png
Administration & Architecture
Reply
  • 1805 Views
  • 1 replies
  • 0 kudos
View Replies
Latest Reply
Debayan
Esteemed Contributor III
  • 0 kudos

Hi, Could you please send the whole screenshot? Is it asking to login to azure and asking Azure's approval or Databricks? 

  • 0 kudos
PetrSindelar
by • New Contributor II
  • 2555 Views
  • 1 replies
  • 1 kudos

Connecting external location from a different tenant in Azure

Hi,we have a setup with 2 different Azure tenants. In tenant A we have a storage account that we want to connect as an external location to a databricks workspace in the tenant B. For that we have established a private endpoint from the storage accou...

Administration & Architecture
Reply
  • 2555 Views
  • 1 replies
  • 1 kudos
View Replies
Latest Reply
Debayan
Esteemed Contributor III
  • 1 kudos

Hi, As cross tenant has to be configured from the Azure side first, can this be verified with the Azure team in-parallel?  https://learn.microsoft.com/en-us/azure/virtual-network-manager/how-to-configure-cross-tenant-portal

  • 1 kudos
OwenDB
by • New Contributor III
  • 5531 Views
  • 5 replies
  • 3 kudos

Spark jobs not starting for saving

I am having issues with a new workspace that has been created and having trouble diagnosing what the issue is.It seems to be related to compute resources, spark and storage.I am an admin and able to create compute resources. Using SQL warehouses I ca...

Administration & Architecture
Reply
  • 5531 Views
  • 5 replies
  • 3 kudos
View Replies
Latest Reply
OwenDB
New Contributor III
  • 3 kudos

ok looks like it might be this for us....You must have a separate pair of host/container subnets for each workspace that you deploy. It is unsupported to share subnets across workspaces or to deploy other Azure resources on the subnets that are used ...

  • 3 kudos
4 More Replies
deanb-everc
by • New Contributor
  • 1230 Views
  • 1 replies
  • 0 kudos

"Forgot my password" does not send mail to users in my organization.

Hi, we are using databricks in my organization. SSO logins stopped working for us about a week ago. As a workaround, we asked our users to use the admin login page, and use their organization passwords. This didn't work because the passwords were inc...

Administration & Architecture
Reply
  • 1230 Views
  • 1 replies
  • 0 kudos
View Replies
Latest Reply
Kaniz_Fatma
Community Manager
  • 0 kudos

Hi @deanb-everc, Please raise a support ticket.

  • 0 kudos
Join 100K+ Data Experts: Register Now & Grow with Us!

Excited to expand your horizons with us? Click here to Register and begin your journey to success!

Already a member? Login and join your local regional user group! If there isn’t one near you, fill out this form and we’ll create one for you to join!

Labels