cancel
Turn on suggestions
Auto-suggest helps you quickly narrow down your search results by suggesting possible matches as you type.
Showing results for 
Search instead for 
Did you mean: 
Help
Data Engineering
Join discussions on data engineering best practices, architectures, and optimization strategies within the Databricks Community. Exchange insights and solutions with fellow data engineers.
cancel
Turn on suggestions
Auto-suggest helps you quickly narrow down your search results by suggesting possible matches as you type.
Showing results for 
Search instead for 
Did you mean: 

Forum Posts

Start a conversation
nistrate
by • New Contributor III
  • 5524 Views
  • 2 replies
  • 3 kudos

Resolved! Revoking cluster creation permission for non-admin users in Databricks workspace

I'm currently building a Databricks workspace for my organization and I would like to revoke the cluster creation permission for all users except admins or a specific group (e.g., Data Analysts). Could someone please guide me on how to achieve this?

Data Engineering
Reply
  • 5524 Views
  • 2 replies
  • 3 kudos
View Replies
Latest Reply
Anonymous
Not applicable
  • 3 kudos

Hi @Nicolae Istrate​ We haven't heard from you since the last response from @Werner Stinckens​ , and I was checking back to see if her suggestions helped you.Or else, If you have any solution, please share it with the community, as it can be helpful ...

  • 3 kudos
1 More Replies
Prank
by • New Contributor III
  • 1192 Views
  • 1 replies
  • 1 kudos

Access DBU used per cluster using within Databricks Clusters

Could it be possible, we can retrieve the DBU's on cluster basis within Databricks Notebook itself?This info we get in the compute tab in Databricks for each cluster as Active DBU/hr.

Data Engineering
Reply
  • 1192 Views
  • 1 replies
  • 1 kudos
View Replies
Latest Reply
Anonymous
Not applicable
  • 1 kudos

It wont be possible to access DBU used per cluster within DB Clusters.

  • 1 kudos
sagiatul
by • New Contributor II
  • 4921 Views
  • 2 replies
  • 3 kudos

Databricks driver logs

I am running jobs on databricks clusters. When the cluster is running I am able to find the executor logs by going to Spark Cluster UI Master dropdown, selecting a worker and going through the stderr logs. However, once the job is finished and cluste...

image
Data Engineering
Reply
  • 4921 Views
  • 2 replies
  • 3 kudos
View Replies
Latest Reply
Anonymous
Not applicable
  • 3 kudos

Hi @Atul Arora​ Thank you for your question! To assist you better, please take a moment to review the answer and let me know if it best fits your needs.Please help us select the best solution by clicking on "Select As Best" if it does.Your feedback w...

  • 3 kudos
1 More Replies
Jfoxyyc
by • Valued Contributor
  • 4724 Views
  • 8 replies
  • 7 kudos

Databricks IDE - Notebooks

With the announcement of the official IDE support for VS Code, does any one know if there's a way to run notebooks in VSC Code on Databricks clusters?https://www.databricks.com/blog/2023/02/14/announcing-a-native-visual-studio-code-experience-for-dat...

Data Engineering
Reply
  • 4724 Views
  • 8 replies
  • 7 kudos
View Replies
Latest Reply
Jfoxyyc
Valued Contributor
  • 7 kudos

There has been no response regarding running notebook cells in VS Code, so no best response.

  • 7 kudos
7 More Replies
AlexDavies
by • Contributor
  • 2672 Views
  • 2 replies
  • 2 kudos

Issue connecting to SQL warehouse spark thrift server

we have a library that allows dotnet applications to talk to databricks clusters (https://github.com/clearbank/SparkSqlClient). This communicates with the clusters over the spark thrift serverAlthough this works great for clusters in the "data scienc...

Data Engineering
Reply
  • 2672 Views
  • 2 replies
  • 2 kudos
View Replies
Latest Reply
AlexDavies
Contributor
  • 2 kudos

I have tried those connection details however it they give me 400 errors when trying to connect directly using the hive thrift server contract (https://github.com/apache/hive/blob/master/service-rpc/if/TCLIService.thrift). I do not get the issues whe...

  • 2 kudos
1 More Replies
antoooks
by • New Contributor III
  • 7122 Views
  • 5 replies
  • 10 kudos

Resolved! Databricks clusters stuck on Pending and Terminating state indefinitely

Hi everyone,Our company is using Databricks on GKE. It works fine until suddenly when we try to create and terminate clusters today, it got stuck on Pending and Terminating state for hours (now more than 6 hours). There is no conclusion can be drawn ...

screenshot
Data Engineering
Reply
  • 7122 Views
  • 5 replies
  • 10 kudos
View Replies
Latest Reply
Databricks_Buil
New Contributor III
  • 10 kudos

Hi @Kurnianto Trilaksono Sutjipto​ : Figured out after multiple connects that This is typically a cloud provider issue. You can file a support ticket if the issue persists.

  • 10 kudos
4 More Replies
Mohit_m
by • Valued Contributor II
  • 10328 Views
  • 1 replies
  • 3 kudos

Resolved! How to Install Python packages from the own artifactory

We have created our own artifactory and we use this to install python dependencies or libraries.We would like to know how we can make use of our own artifactory to install dependencies or libraries on Databricks clusters..

Data Engineering
Reply
  • 10328 Views
  • 1 replies
  • 3 kudos
View Replies
Latest Reply
Mohit_m
Valued Contributor II
  • 3 kudos

For private repos, you can find some good examples herehttps://kb.databricks.com/clusters/install-private-pypi-repo.htmlhttps://towardsdatascience.com/install-custom-python-libraries-from-private-pypi-on-databricks-6a7669f6e6fd

  • 3 kudos
User16826987838
by • Contributor
  • 1206 Views
  • 2 replies
  • 0 kudos

Looking for information on security design on how JDBC connections to clusters function

I am looking for more information around the security design around how JDBC connections to clusters function:What security controls are in operation to safeguard the Databricks clusters?Is the API gateway abstracted from the Databricks cluster in th...

Data Engineering
Reply
  • 1206 Views
  • 2 replies
  • 0 kudos
View Replies
Latest Reply
Albina228
New Contributor II
  • 0 kudos

In fact, I have no idea what kind of design we are talking about, it causes associations with Cloud Ceilings

  • 0 kudos
1 More Replies
thomasthomas
by • New Contributor II
  • 2254 Views
  • 4 replies
  • 0 kudos

Resolved! Customer deployment

Hi,I have a bunch of scripts in Databricks that perform a decent amount of data-wrangling. All of these scripts contain sensitive information and I have no intention of making them public.I would like to provide a service to my customers - so they ca...

Data Engineering
Reply
  • 2254 Views
  • 4 replies
  • 0 kudos
View Replies
Latest Reply
Atanu
Databricks Employee
  • 0 kudos

@Tamas D​  I understood your concern. For cluster creation in different subscription I think that's by design at this moment. But I would like to request you to add your use case to https://feedback.azure.com/d365community/forum/2efba7dc-ef24-ec11-b6...

  • 0 kudos
3 More Replies
Mohit_m
by • Valued Contributor II
  • 835 Views
  • 1 replies
  • 2 kudos

docs.databricks.com

Are EBS volumes used by Databricks Clusters are encrypted especially the root volumes

Data Engineering
Reply
  • 835 Views
  • 1 replies
  • 2 kudos
View Replies
Latest Reply
Mohit_m
Valued Contributor II
  • 2 kudos

Yes these EBS volumes are encrypted. Earlier root volume encryptions were not supported but recently this encryption is also enabled (since Apr, 2021)please find more details on the below docs pagehttps://docs.databricks.com/clusters/configure.html#e...

  • 2 kudos
aladda
by • Databricks Employee
  • 1062 Views
  • 1 replies
  • 0 kudos

Can i run data engineering/data transformation jobs using databricks SQL endpoints or are databricks clusters (interactive or jobs) the recommended way to do this?

Data Engineering
Reply
  • 1062 Views
  • 1 replies
  • 0 kudos
View Replies
Latest Reply
Ryan_Chynoweth
Esteemed Contributor
  • 0 kudos

Generally, interactive clusters and jobs are better suited for data engineering and transformations as they support more than just SQL. However, if you are using pure SQL, then endpoints can be used for data transformations. All of the Spark SQL fun...

  • 0 kudos
Labels