cancel
Showing results for 
Search instead for 
Did you mean: 
Administration & Architecture
Explore discussions on Databricks administration, deployment strategies, and architectural best practices. Connect with administrators and architects to optimize your Databricks environment for performance, scalability, and security.
cancel
Showing results for 
Search instead for 
Did you mean: 
Data + AI Summit 2024 - Data Lakehouse Architecture


Forum Posts

Carson
by New Contributor
  • 242 Views
  • 0 replies
  • 0 kudos

How to monitor serverless compute usage in real time

Hello, I'm using Databricks Connect to connect a dash app to my Databricks account. My use case is similar to this example: https://github.com/databricks-demos/dbconnect-examples/tree/main/python/PlotlyI've been able to get everything configured and ...

  • 242 Views
  • 0 replies
  • 0 kudos
AClarkson
by New Contributor II
  • 6929 Views
  • 3 replies
  • 0 kudos

AWS Secrets Manager access

HiWe're in the process of moving over from Databricks in Azure to AWS.I am trying to establish a method of accessing secrets from AWS Secrets Manager (we were using Azure KeyVault) and understand this can be done with boto as suggested from AWS.We ha...

  • 6929 Views
  • 3 replies
  • 0 kudos
Latest Reply
AClarkson
New Contributor II
  • 0 kudos

I should add I'm running this on a shared cluster and therefore suspect the following constraints are impeding me:Taken from:  https://docs.databricks.com/en/clusters/configure.htmlCannot connect to the instance metadata service (IMDS), other EC2 ins...

  • 0 kudos
2 More Replies
Mathias_Peters
by Contributor
  • 336 Views
  • 1 replies
  • 0 kudos

Databricks Terraform Module for reading the current clusters times out

Hi, we are managing our Databricks Workspaces entirely using terraform. As of today, the module to read the current clusters fails with a timeout. We use this:data "databricks_clusters" "this" { }This has worked flawlessly before. However, now we are...

Mathias_Peters_0-1723728679046.png
  • 336 Views
  • 1 replies
  • 0 kudos
Latest Reply
Mathias_Peters
Contributor
  • 0 kudos

Correction of my previous post: The error occurs when bumping the provider version to 1.50.0. It works fine on 1.49.0

  • 0 kudos
mo_moattar
by New Contributor III
  • 204 Views
  • 0 replies
  • 0 kudos

Would it be great if the job workflow supports running docker-based tasks

The current workflow function in Databricks gives a series of options such as DLT, Dbt, python scripts, python files, JAR, etc. It would be good to add a docker file to that and simplify the development process a lot, especially on the unit and integ...

  • 204 Views
  • 0 replies
  • 0 kudos
camilo_s
by Contributor
  • 373 Views
  • 0 replies
  • 0 kudos

Programmatically setting tags for securables

Unity Catalog securable objects can be tagged with key value pairs: https://learn.microsoft.com/en-us/azure/databricks/database-objects/tagsIs it possible tag objects via REST API calls?I initially thought any Unity Catalog resource in the Databricks...

  • 373 Views
  • 0 replies
  • 0 kudos
slloyd
by New Contributor
  • 465 Views
  • 0 replies
  • 0 kudos

client.openSession() : TypeError: Cannot read properties of undefined (reading '0')

I am using the Databricks SQL Driver for Node.js to create an endpoint that queries a Databricks database following the guide here Databricks SQL Driver for Node.js | Databricks on AWS . This code was working previously but now I am getting a TypeErr...

  • 465 Views
  • 0 replies
  • 0 kudos
m997al
by Contributor III
  • 829 Views
  • 2 replies
  • 1 kudos

Azure Databricks SCIM provisioning configured, working...but what happens when user is deleted?

Hi - I have Azure Databricks with Unity Catalog, and the SCIM Connector provisioning up and working.Due to a misunderstanding on my part, I deleted some individual users (using the Databricks admin console) thinking that I was deleting individual acc...

  • 829 Views
  • 2 replies
  • 1 kudos
Latest Reply
m997al
Contributor III
  • 1 kudos

This was resolved...but why it was resolved remains a mystery.  We simply re-provisioned by turning off and then on the SCIM provisioning in the Azure portal for the Databricks SCIM connector (which is supposed to cause a full re-provisioning, but di...

  • 1 kudos
1 More Replies
MichaelO
by New Contributor III
  • 451 Views
  • 1 replies
  • 0 kudos

Terraform databricks new feature

I recently saw a new feature in databricks that allows to set an upper limit of run time for jobs, so it fails when the limit is breached. See image below:Are we able to terraform code this new feature?

MichaelO_0-1723549982471.png
  • 451 Views
  • 1 replies
  • 0 kudos
Latest Reply
Rjdudley
New Contributor III
  • 0 kudos

Isn't that just timeout_seconds?That UI element looks like it corresponds to that property in the REST API (https://docs.databricks.com/api/azure/workspace/jobs/create#timeout_seconds), and that same top-level argument in Terraform, https://registry....

  • 0 kudos
Olex
by New Contributor II
  • 2533 Views
  • 5 replies
  • 1 kudos

Unable to login accounts.azuredatabricks.net and create admin account

Hello everyoneI'm trying to setup my admin account in Databricks Admin Console and am following these steps:https://learn.microsoft.com/en-us/azure/databricks/administration-guide/#--what-are-account-adminsHowever, when I access accounts.azuredatabri...

  • 2533 Views
  • 5 replies
  • 1 kudos
Latest Reply
Olex
New Contributor II
  • 1 kudos

Hi everyoneI asked that question a year and somehow it'd been solved.Not sure however if it's related to generic email providers, but I have multiple registered accounts, gmail and corporate one. For databricks accademy, I'm using gmail account, whic...

  • 1 kudos
4 More Replies
shubhdhanuka
by New Contributor
  • 496 Views
  • 1 replies
  • 0 kudos

Account Access token

Can you please help me with how to authenticate account level rest apis?curl --request GET https://accounts.cloud.databricks.com/api/2.0/accounts/{account_id}/usageI use Permanent Access Token to authenticate workspace APIs but it's not working for w...

  • 496 Views
  • 1 replies
  • 0 kudos
Latest Reply
szymon_dybczak
Contributor III
  • 0 kudos

Hi @shubhdhanuka ,To get access to account API you cannot use personal access token. Instead, you must use the Microsoft Entra ID tokens of Azure Databricks account admins. Azure Databricks account admins can be users or service principals.Below you ...

  • 0 kudos
jenshumrich
by Contributor
  • 1229 Views
  • 3 replies
  • 2 kudos

Databricks UI error

I receive this respone when trying to download a csv file: adb-737684297741988.8.azuredatabricks.net/ajax-api/2.0/workspace/export?path=/Repos/jhu_admin%40siemensgamesa.com/sgre-dfp-lifestrategy-python/cpo/output/Rattlesnake Flats/Rdata/AssetCheck_Ra...

jenshumrich_0-1723198951857.png
  • 1229 Views
  • 3 replies
  • 2 kudos
Latest Reply
jenshumrich
Contributor
  • 2 kudos

@szymon_dybczak I am using the GUI (see the screenshot in the original post): File marked -> Three dots -> Export

  • 2 kudos
2 More Replies
dprutean
by New Contributor III
  • 1181 Views
  • 1 replies
  • 0 kudos

JDBC Error with OpenJDK 21

Connecting to Azure Databricks using latest JDBC driver and OpenJDK 21 I got this error. I already set:--add-opens=java.base/java.nio=ALL-UNNAMED-Dio.netty.tryReflectionSetAccessible=true java.sql.SQLException: [Databricks][JDBCDriver](500618) Error ...

  • 1181 Views
  • 1 replies
  • 0 kudos
Latest Reply
151640
New Contributor III
  • 0 kudos

Version 02.06.38.1068Using SQLSquirrel with ibm-semeru-certified-17-jdk (17.0.11+9)With JRE options --add-opens=jdk.unsupported/sun.misc=ALL-UNNAMED failsShould be  --add-opens=java.base/java.nio=ALL-UNNAMEDcaused by: java.lang.reflect.InaccessibleOb...

  • 0 kudos
renancy
by New Contributor III
  • 283 Views
  • 0 replies
  • 0 kudos

Using Databricks CLI for generating Notebooks not supported or not implemented

Hi I'm a Data engineer and recently developed a Notebook analytics template for general purposes that I would like to be the standard on my company. Continuing, I created another notebook with a text widget that uses the user input to map the folder ...

  • 283 Views
  • 0 replies
  • 0 kudos
mr_elastic
by New Contributor III
  • 446 Views
  • 1 replies
  • 7 kudos

Databricks Unity Catalog Bug - Reset of Network Connectivity Configuration not possible

The following use case is strange regarding the Network Connectivity Configuration (NCC):I create a Workspace (the NCC is empty)I create a NCCI attach the NCC to the WorkspaceI want to remove the NCC from the Workspace -> not possibleTherefore, I can...

  • 446 Views
  • 1 replies
  • 7 kudos
Latest Reply
rcbiczok
New Contributor II
  • 7 kudos

We faced the very same issue in our Tenant.

  • 7 kudos

Connect with Databricks Users in Your Area

Join a Regional User Group to connect with local Databricks users. Events will be happening in your city, and you won’t want to miss the chance to attend and share knowledge.

If there isn’t a group near you, start one and help create a community that brings people together.

Request a New Group
Labels