cancel
Showing results for 
Search instead for 
Did you mean: 
Administration & Architecture
Explore discussions on Databricks administration, deployment strategies, and architectural best practices. Connect with administrators and architects to optimize your Databricks environment for performance, scalability, and security.
cancel
Showing results for 
Search instead for 
Did you mean: 
Data + AI Summit 2024 - Data Lakehouse Architecture


Forum Posts

Joaquim
by New Contributor II
  • 5421 Views
  • 3 replies
  • 0 kudos

New admin question: How do you enable R on a existing cluster?

Hello Community. I have a user trying to use R and receive the error message illustrated on the attachment. I can't seem to find correct documentation on enabling R on an existing cluster. Would anyone be able to point me in the right direction? Than...

  • 5421 Views
  • 3 replies
  • 0 kudos
Latest Reply
Aviral-Bhardwaj
Esteemed Contributor III
  • 0 kudos

This will help you

  • 0 kudos
2 More Replies
saikrishna3390
by New Contributor II
  • 2251 Views
  • 2 replies
  • 1 kudos

Resolved! How do we get user list who accessed specific table or view in Unity catalog for last 6 months

We have a business use case where we want to track users who accessed a specific table in Unity catalog for last 6 months . Is there a way where we can pull this data ? 

  • 2251 Views
  • 2 replies
  • 1 kudos
Latest Reply
Aviral-Bhardwaj
Esteemed Contributor III
  • 1 kudos

yes system table will have all details

  • 1 kudos
1 More Replies
madhura
by New Contributor II
  • 1260 Views
  • 3 replies
  • 0 kudos

Rest endpoint for data bricks audit logs

I am trying to find official documentation link to get audit logs of data bricks. unable to find it. referred on.https://docs.databricks.com/en/administration-guide/account-settings/audit-logs.htmlhttps://docs.databricks.com/api/workspace/introductio...

  • 1260 Views
  • 3 replies
  • 0 kudos
Latest Reply
Yeshwanth
Databricks Employee
  • 0 kudos

@madhura  I could not find any endpoint that can be used to get the Audit logs. However, you can enable system tables in your workspace and try to read the data just like you read from any other table. Please check this to enable system tables: https...

  • 0 kudos
2 More Replies
Priyam1
by New Contributor III
  • 1660 Views
  • 1 replies
  • 0 kudos

Databricks Job alerts

I'm currently running jobs on job clusters and would like these jobs to time out after 168 hours (7 days), at which point a new job cluster will be assigned. This timeout is specifically to ensure that jobs don't run on the same cluster for too long,...

  • 1660 Views
  • 1 replies
  • 0 kudos
Latest Reply
Yeshwanth
Databricks Employee
  • 0 kudos

@Priyam1 Good day!Based on the information provided, it seems that we do not have a direct way to mute notifications for timed-out jobs while still receiving alerts for job failures. You can reduce the number of notifications sent by filtering out no...

  • 0 kudos
mathiaskvist
by New Contributor III
  • 5247 Views
  • 7 replies
  • 1 kudos

Resolved! Valid Workspace Conf keys

HiI'm trying to automate the configuration of Admin Settings of our Databricks Workspace using Terraform. However identifying the correct config keys is very difficult.Databricks exposes a Workspace Conf API (Enable/disable features | Workspace Conf ...

  • 5247 Views
  • 7 replies
  • 1 kudos
Latest Reply
Alexis_Chicoine
New Contributor II
  • 1 kudos

I wanted to know the key for Store interactive notebook results in customer account.It's not ideal but by using the browser dev tools you can find out what it is by looking at the network activity after toggling it in the UI.  

  • 1 kudos
6 More Replies
m997al
by Contributor III
  • 3507 Views
  • 6 replies
  • 3 kudos

Azure Databricks with standard private link cluster event log error: "Metastore down"...

We have Azure Databricks with standard private link (back-end and front-end private link).We are able to successfully attach a Databricks workspace to the Databricks metastore (ADLS Gen2 storage).However, when trying to create tables in a catalog in ...

  • 3507 Views
  • 6 replies
  • 3 kudos
Latest Reply
CharlesWoo
New Contributor II
  • 3 kudos

can confirm that the approach will solve your error. Ran into a similar issue a while back.

  • 3 kudos
5 More Replies
aockenden
by New Contributor III
  • 1300 Views
  • 4 replies
  • 0 kudos

Secrets ACL API Behavior Change

Hey all,Has the behavior of the Secrets ACL API changed over the last 24 hours? With no code changes on our scope-deployment pipeline, I am suddenly getting strange errors back from this endpoint.Anybody else noticing a change?Thanks,Alex

  • 1300 Views
  • 4 replies
  • 0 kudos
Latest Reply
aockenden
New Contributor III
  • 0 kudos

Idk, I control the resource group myself and I don't remember ever granting or revoking contributor roles on that RG for any of these users which are now suddenly throwing errors. Interesting to see that line from the docs... I wonder if that was alw...

  • 0 kudos
3 More Replies
Fresher
by New Contributor II
  • 2626 Views
  • 4 replies
  • 1 kudos

Databricks is taking too long to run a query

I ran a simple query. It is taking too much time and it doesn't stop running. I am the only one who is using cluster. It's happening to every notebook in the workspace.

  • 2626 Views
  • 4 replies
  • 1 kudos
Latest Reply
Lakshay
Databricks Employee
  • 1 kudos

Do you see spark jobs running in Spark UI ?

  • 1 kudos
3 More Replies
harripy
by New Contributor III
  • 1452 Views
  • 1 replies
  • 0 kudos

Proper way to collect Statement ID from JDBC Connection

Hi, We are executing DML calls on Databricks SQL Warehouse programmatically, with Java and Python.There can be thousands of executions running on daily level, so in case of an error occurs, it would be very beneficial to spot the Statement ID of the ...

  • 1452 Views
  • 1 replies
  • 0 kudos
Latest Reply
" src="" />
This widget could not be displayed.
This widget could not be displayed.
This widget could not be displayed.
  • 0 kudos

This widget could not be displayed.
Hi, We are executing DML calls on Databricks SQL Warehouse programmatically, with Java and Python.There can be thousands of executions running on daily level, so in case of an error occurs, it would be very beneficial to spot the Statement ID of the ...

This widget could not be displayed.
  • 0 kudos
This widget could not be displayed.
harripy
by New Contributor III
  • 2201 Views
  • 2 replies
  • 1 kudos

Resolved! Native service principal support in JDBC/ODBC drivers

Read from Databricks integration best practises about the native support for Service Principal authentication on JDBC/ODBC drivers. The timetable mentioned for this was "expected to land in 2023", is this referring to the https://docs.databricks.com/...

  • 2201 Views
  • 2 replies
  • 1 kudos
Latest Reply
daniel_sahal
Esteemed Contributor
  • 1 kudos

@harripy As it says in the documentation, JDBC driver 2.6.36 and above supports Azure Databricks OAuth secrets for OAuth M2M or OAuth 2.0 client credentials authentication. Microsoft Entra ID secrets are not supported.https://learn.microsoft.com/en-u...

  • 1 kudos
1 More Replies
Swoops
by New Contributor II
  • 1728 Views
  • 1 replies
  • 2 kudos

Resolved! What happens to the Workspace when deleting the user that created that workspace

Hi,Background:In Azure-Datebricks:We have a personal account that created a workspace, this workspace is our main workspace. The users Microsoft Office account is already deleted but the user still exist in Databricks.Question 1 is:What happens to th...

  • 1728 Views
  • 1 replies
  • 2 kudos
Latest Reply
arpit
Databricks Employee
  • 2 kudos

What happens to this workspace if we delete the User from workspace? Will the Workspace be deleted? → Workspace will not get deleted. It will continue to function.Is it possible to change Workspace own to another account i.e. a service account. If so...

  • 2 kudos
Dicer
by Valued Contributor
  • 2543 Views
  • 1 replies
  • 1 kudos

Resolved! Import folder (no .whl or .jar files) and run the `python3 setup.py bdist_wheel` for lib install

I want to import the ibapi python module in Azure Databricks Notebook.Before this, I downloaded the the TWS API folder from https://interactivebrokers.github.io/# I need to go through the following steps to install the API:Download and install TWS Ga...

  • 2543 Views
  • 1 replies
  • 1 kudos
Latest Reply
arpit
Databricks Employee
  • 1 kudos

You can try to upload the folder in the workspace location and try to cd in the desired folder and try to install in via notebook. But it would be a notebook scope installation. If you are looking for a cluster scoped installation then you would need...

  • 1 kudos
Datastoryteller
by New Contributor II
  • 1328 Views
  • 3 replies
  • 0 kudos

Permission denied using patchelf

Hello all I would like to run a Python script on a Shared cluster: The Python script tries to do, under the hood, a call to `patchelf` utility, in order to set a r-path. Something along those lines:execute('patchelf',['--set-rpath', rpath, lib])The p...

Screenshot 2024-04-17 at 16.00.03.png
  • 1328 Views
  • 3 replies
  • 0 kudos
Latest Reply
Datastoryteller
New Contributor II
  • 0 kudos

I basically followed this tutorial https://learn.microsoft.com/en-gb/azure/databricks/data-governance/unity-catalog/get-started and it seemed to be working, so I guess Unity Catalog is enabled?

  • 0 kudos
2 More Replies
phguk
by New Contributor III
  • 2245 Views
  • 1 replies
  • 0 kudos

Why does use of Azure SSO require Databricks PAT enabled ?

My org uses Databricks and SSO. We are keen to disable the use of PAT but have noticed that when it's disabled, we're not able to use SSO. May I ask why does SSO have a dependency on PATs [arguably they are two distinct authentication methods] ?Also,...

  • 2245 Views
  • 1 replies
  • 0 kudos
Latest Reply
" src="" />
This widget could not be displayed.
This widget could not be displayed.
This widget could not be displayed.
  • 0 kudos

This widget could not be displayed.
My org uses Databricks and SSO. We are keen to disable the use of PAT but have noticed that when it's disabled, we're not able to use SSO. May I ask why does SSO have a dependency on PATs [arguably they are two distinct authentication methods] ?Also,...

This widget could not be displayed.
  • 0 kudos
This widget could not be displayed.

Connect with Databricks Users in Your Area

Join a Regional User Group to connect with local Databricks users. Events will be happening in your city, and you won’t want to miss the chance to attend and share knowledge.

If there isn’t a group near you, start one and help create a community that brings people together.

Request a New Group
Labels