cancel
Showing results for 
Search instead for 
Did you mean: 
Administration & Architecture
Explore discussions on Databricks administration, deployment strategies, and architectural best practices. Connect with administrators and architects to optimize your Databricks environment for performance, scalability, and security.
cancel
Showing results for 
Search instead for 
Did you mean: 
Data + AI Summit 2024 - Data Lakehouse Architecture


Forum Posts

_rafa
by New Contributor
  • 1470 Views
  • 1 replies
  • 0 kudos

Azure SP OAuth wont generate token

Hi all,Hopefully someone has had this issue and fixed it before. No matter how many times I try, no token is generated (yes, i am in a premium tier). Any ideas? I tried using the API but I got this error that apparently is because I am in Azure. for ...

_rafa_0-1702294084413.png _rafa_1-1702294208009.png _rafa_0-1702294866329.png _rafa_1-1702294945196.png
  • 1470 Views
  • 1 replies
  • 0 kudos
Latest Reply
Yeshwanth
Honored Contributor
  • 0 kudos

Hello @_rafa  ,Currently, we do not have an API similar to the "https://<databricks-instance>/api/2.0/token-management/on-behalf-of/tokens" in azure as Azure supports the service principle authentication by default. Please authenticate to normal PAT ...

  • 0 kudos
Wojciech_BUK
by Valued Contributor III
  • 2573 Views
  • 1 replies
  • 0 kudos

Azure Databricks SQL Warehouse tags not being populated

In Azure cost monitor I am trying to get list of my cluster names and cost associated to it.It works fine with job cluster and all purpos clusters but not for SQL Warehouses.When you look at below example, where I am querying only for my SQL Warehous...

Wojciech_BUK_0-1703773554117.png Wojciech_BUK_1-1703773672483.png
Administration & Architecture
azure
cost
monitoring
SQL warehouse
tags
  • 2573 Views
  • 1 replies
  • 0 kudos
Latest Reply
Yeshwanth
Honored Contributor
  • 0 kudos

@Wojciech_BUK I would recommend you get in touch with the Azure Pricing team once to check this in a great deatil.

  • 0 kudos
dbx_8451
by New Contributor II
  • 940 Views
  • 1 replies
  • 0 kudos

How to delete or update cluster-scoped init scripts

We are working on to deprecate the dbfs based cluster-scoped init scripts and replace them with workspace based, therefore finding a way to delete the dbfs cluster-scoped init scripts from all the clusters that are running in our environment using RE...

  • 940 Views
  • 1 replies
  • 0 kudos
Latest Reply
arpit
Valued Contributor
  • 0 kudos

@dbx_8451 There is no direct way for doing it, you may need to automate using the list Cluster API.

  • 0 kudos
bpark
by New Contributor
  • 1556 Views
  • 1 replies
  • 0 kudos

No enable option for Customize containers with Databricks Container Services on Azure and GCP

Customize containers with Databricks Container ServicesI want to create a spark cluster with the hail container [1]. But there is no Enable Container Services on the Workspace configuration for Azure Databricks and GCP Databricks.The API does not ret...

  • 1556 Views
  • 1 replies
  • 0 kudos
Latest Reply
arpit
Valued Contributor
  • 0 kudos

Did you tested out using the cluster API ?

  • 0 kudos
alesventus
by Contributor
  • 1302 Views
  • 1 replies
  • 0 kudos

Change users status in account console

One of our users has status = Inactive in Admin setting. I tried to set it to active using rest api and the documentation but seems like api is not changing status.I tested api and I can list users so api setting are ok.api request: import requests i...

  • 1302 Views
  • 1 replies
  • 0 kudos
Latest Reply
arpit
Valued Contributor
  • 0 kudos

@alesventus Can you test the same using CLI or may be try to make the status as inactive for a dummy user? Just want to validate if the API is actually working for you or not.

  • 0 kudos
ianc
by New Contributor
  • 734 Views
  • 1 replies
  • 0 kudos

spark.databricks documentation

I cannot find any documentation related to the spark.databricks.* I was able to find the spark related documentation but it does not contain any information on possible properties or arguments for spark.databricks in particular. Thank you!

  • 734 Views
  • 1 replies
  • 0 kudos
Latest Reply
arpit
Valued Contributor
  • 0 kudos

Hello @ianc , Databricks have documentation for the product feature and its underlying spark properties(as applicable). If you are looking for one place documentation for all the spark.databricks.*, it is not available. hOwever we can help to find pr...

  • 0 kudos
ashdam
by New Contributor III
  • 968 Views
  • 1 replies
  • 0 kudos

Installing databricks CLI on windows . sh ??¿?

I am following databricks documentaiton for installing the cli on windows and i found that uses sh command which of course doesnt work.My windows in admin mode says cannot find "sh" application. I could install cgwyn and tools like that but since the...

ashdam_0-1698741976775.png ashdam_1-1698742182892.png
  • 968 Views
  • 1 replies
  • 0 kudos
Latest Reply
arpit
Valued Contributor
  • 0 kudos

Here is the doc which you can refer:AWS, GCP, and Azure 

  • 0 kudos
anurag_dev
by New Contributor
  • 1165 Views
  • 1 replies
  • 0 kudos

getting alerts in AWS UnauthorizedAccess:IAMUser/InstanceCredentialExfiltration.InsideAWS

Hi folks, I have been using IAM instance profile as a credentials provider to Databricks workspace. I am getting this error "Credentials created exclusively for an EC2 instance using instance role <role-arn> have been used from a remote AWS account 4...

  • 1165 Views
  • 1 replies
  • 0 kudos
Latest Reply
Yeshwanth
Honored Contributor
  • 0 kudos

Hi @anurag_dev  Databricks runs a commit service that coordinates writes to Amazon S3 from multiple clusters. This service runs in the Databricks control plane. (441371677306)The S3 commit service helps guarantee consistency of writes across multiple...

  • 0 kudos
bas
by New Contributor III
  • 7373 Views
  • 9 replies
  • 2 kudos

Resolved! Adding a service principal to workspace fails

Trying to add an Azure Service Principal to a workspace fails, whether via the web UI or the databricks cli. With the databricks, the console gives this:databricks service-principals -p ************* create --application-id ********-****-****-****-**...

  • 7373 Views
  • 9 replies
  • 2 kudos
Latest Reply
brickmaker
New Contributor II
  • 2 kudos

Same issue in AWS Databricks.

  • 2 kudos
8 More Replies
msserpa
by New Contributor II
  • 3001 Views
  • 2 replies
  • 0 kudos

Can we turn off Playground & Marketplace for some users?

Hi Everyone,Hope all is well with you!I'm reaching out to the community for some advice on customizing our workspace settings. Specifically, I have two things I'm trying to figure out:Disabling the "Playground" Option: Is there a way to turn off the ...

169300_1-1704556176241.png
Administration & Architecture
Databricks
Marketplace
Playground
  • 3001 Views
  • 2 replies
  • 0 kudos
Latest Reply
Wing139
New Contributor II
  • 0 kudos

how can we turn on the playground option, i cannot find it

  • 0 kudos
1 More Replies
sl79
by New Contributor II
  • 1936 Views
  • 1 replies
  • 1 kudos

Unable to start a SQL cluster

We have a GCP databricks hosted deployment. last night our SQL warehouse failed to start and now just sits with this message:"Cluster Start-up Delayed. Please wait while we continue to try and start the cluster. No action is required from you"It didn...

  • 1936 Views
  • 1 replies
  • 1 kudos
Latest Reply
jose_gonzalez
Moderator
  • 1 kudos

Could you share the driver logs if you have them??

  • 1 kudos
Learnit
by New Contributor II
  • 2643 Views
  • 1 replies
  • 0 kudos

Managing databricks workspace permissions

I need assistance with writing API/Python code to manage a Databricks workspace permissions database(unity catalog). The task involves obtaining a list of workspace details from the account console, which includes various details like Workspace name,...

  • 2643 Views
  • 1 replies
  • 0 kudos
Latest Reply
Kaniz_Fatma
Community Manager
  • 0 kudos

Hi @Learnit, Here’s a high-level overview of the steps you might take:   Obtain Workspace Details: You can use the Databricks REST API to obtain workspace details. The API provides various endpoints to manage access for different objects and users. M...

  • 0 kudos
pawelzak
by New Contributor III
  • 1031 Views
  • 2 replies
  • 0 kudos

Dashboard programmatical update

Hi,The question that I would like to answer is: How can one replicate dashboards across different environments? Ideally I would like to be able to store the definition of a dashboard in my repository and upload it to Databricks with code: this could ...

  • 1031 Views
  • 2 replies
  • 0 kudos
Latest Reply
shan_chandra
Esteemed Contributor
  • 0 kudos

@pawelzak  - The Dashboards can not be exported using workspace /export API. 

  • 0 kudos
1 More Replies
m997al
by Contributor III
  • 3215 Views
  • 4 replies
  • 0 kudos

Azure Databricks with metastore, cannot create managed table

We have set up Azure Databricks with Unity Catalog (metastore) in an ADLS Gen2 storage account.Used Managed Identity (Databricks Access Connector) for connection from workspace(s) to ADLS Gen2ADLS Gen2 storage account has Storage Blob Data Owner and ...

m997al_0-1697803440437.png
  • 3215 Views
  • 4 replies
  • 0 kudos
Latest Reply
Kris2
New Contributor II
  • 0 kudos

@m997al Were you able to resolve this issue? I have same issue as you described. I am able to upload into Volumes but not able to create managed tables in ADLS metastore storage account..

  • 0 kudos
3 More Replies
Matt4
by New Contributor III
  • 2846 Views
  • 2 replies
  • 0 kudos

Resolved! System tables in Azure with private link

Hi,Is there any restriction on what is able to be enabled with System tables when using Azure Databricks with a private link connection?  And is there a way around the restriction (that is secure)?In a public network I am able to enable the following...

  • 2846 Views
  • 2 replies
  • 0 kudos
Latest Reply
Matt4
New Contributor III
  • 0 kudos

Turns out it was an issue with Microsoft somewhere, they've now fixed it and all the system schemas have been enabled that I can enable"schemas":[{"schema":"operational_data","state":"UNAVAILABLE"},{"schema":"access","state":"ENABLE_COMPLETED"},{"sch...

  • 0 kudos
1 More Replies

Connect with Databricks Users in Your Area

Join a Regional User Group to connect with local Databricks users. Events will be happening in your city, and you won’t want to miss the chance to attend and share knowledge.

If there isn’t a group near you, start one and help create a community that brings people together.

Request a New Group
Labels