cancel
Showing results for 
Search instead for 
Did you mean: 
Data Engineering
cancel
Showing results for 
Search instead for 
Did you mean: 

Forum Posts

nistrate
by New Contributor II
  • 2487 Views
  • 2 replies
  • 3 kudos

Resolved! Revoking cluster creation permission for non-admin users in Databricks workspace

I'm currently building a Databricks workspace for my organization and I would like to revoke the cluster creation permission for all users except admins or a specific group (e.g., Data Analysts). Could someone please guide me on how to achieve this?

  • 2487 Views
  • 2 replies
  • 3 kudos
Latest Reply
Anonymous
Not applicable
  • 3 kudos

Hi @Nicolae Istrate​ We haven't heard from you since the last response from @Werner Stinckens​ , and I was checking back to see if her suggestions helped you.Or else, If you have any solution, please share it with the community, as it can be helpful ...

  • 3 kudos
1 More Replies
Prank
by New Contributor III
  • 664 Views
  • 1 replies
  • 1 kudos

Access DBU used per cluster using within Databricks Clusters

Could it be possible, we can retrieve the DBU's on cluster basis within Databricks Notebook itself?This info we get in the compute tab in Databricks for each cluster as Active DBU/hr.

  • 664 Views
  • 1 replies
  • 1 kudos
Latest Reply
Anonymous
Not applicable
  • 1 kudos

It wont be possible to access DBU used per cluster within DB Clusters.

  • 1 kudos
sagiatul
by New Contributor II
  • 1455 Views
  • 2 replies
  • 3 kudos

Databricks driver logs

I am running jobs on databricks clusters. When the cluster is running I am able to find the executor logs by going to Spark Cluster UI Master dropdown, selecting a worker and going through the stderr logs. However, once the job is finished and cluste...

image
  • 1455 Views
  • 2 replies
  • 3 kudos
Latest Reply
Anonymous
Not applicable
  • 3 kudos

Hi @Atul Arora​ Thank you for your question! To assist you better, please take a moment to review the answer and let me know if it best fits your needs.Please help us select the best solution by clicking on "Select As Best" if it does.Your feedback w...

  • 3 kudos
1 More Replies
Jfoxyyc
by Valued Contributor
  • 1925 Views
  • 8 replies
  • 7 kudos

Databricks IDE - Notebooks

With the announcement of the official IDE support for VS Code, does any one know if there's a way to run notebooks in VSC Code on Databricks clusters?https://www.databricks.com/blog/2023/02/14/announcing-a-native-visual-studio-code-experience-for-dat...

  • 1925 Views
  • 8 replies
  • 7 kudos
Latest Reply
Jfoxyyc
Valued Contributor
  • 7 kudos

There has been no response regarding running notebook cells in VS Code, so no best response.

  • 7 kudos
7 More Replies
AlexDavies
by Contributor
  • 1091 Views
  • 2 replies
  • 2 kudos

Issue connecting to SQL warehouse spark thrift server

we have a library that allows dotnet applications to talk to databricks clusters (https://github.com/clearbank/SparkSqlClient). This communicates with the clusters over the spark thrift serverAlthough this works great for clusters in the "data scienc...

  • 1091 Views
  • 2 replies
  • 2 kudos
Latest Reply
AlexDavies
Contributor
  • 2 kudos

I have tried those connection details however it they give me 400 errors when trying to connect directly using the hive thrift server contract (https://github.com/apache/hive/blob/master/service-rpc/if/TCLIService.thrift). I do not get the issues whe...

  • 2 kudos
1 More Replies
antoooks
by New Contributor III
  • 3831 Views
  • 6 replies
  • 10 kudos

Resolved! Databricks clusters stuck on Pending and Terminating state indefinitely

Hi everyone,Our company is using Databricks on GKE. It works fine until suddenly when we try to create and terminate clusters today, it got stuck on Pending and Terminating state for hours (now more than 6 hours). There is no conclusion can be drawn ...

screenshot
  • 3831 Views
  • 6 replies
  • 10 kudos
Latest Reply
Databricks_Buil
New Contributor III
  • 10 kudos

Hi @Kurnianto Trilaksono Sutjipto​ : Figured out after multiple connects that This is typically a cloud provider issue. You can file a support ticket if the issue persists.

  • 10 kudos
5 More Replies
Mohit_m
by Valued Contributor II
  • 6181 Views
  • 1 replies
  • 2 kudos

Resolved! How to Install Python packages from the own artifactory

We have created our own artifactory and we use this to install python dependencies or libraries.We would like to know how we can make use of our own artifactory to install dependencies or libraries on Databricks clusters..

  • 6181 Views
  • 1 replies
  • 2 kudos
Latest Reply
Mohit_m
Valued Contributor II
  • 2 kudos

For private repos, you can find some good examples herehttps://kb.databricks.com/clusters/install-private-pypi-repo.htmlhttps://towardsdatascience.com/install-custom-python-libraries-from-private-pypi-on-databricks-6a7669f6e6fd

  • 2 kudos
User16826987838
by Contributor
  • 657 Views
  • 2 replies
  • 0 kudos

Looking for information on security design on how JDBC connections to clusters function

I am looking for more information around the security design around how JDBC connections to clusters function:What security controls are in operation to safeguard the Databricks clusters?Is the API gateway abstracted from the Databricks cluster in th...

  • 657 Views
  • 2 replies
  • 0 kudos
Latest Reply
Albina228
New Contributor II
  • 0 kudos

In fact, I have no idea what kind of design we are talking about, it causes associations with Cloud Ceilings

  • 0 kudos
1 More Replies
thomasthomas
by New Contributor II
  • 1103 Views
  • 4 replies
  • 0 kudos

Resolved! Customer deployment

Hi,I have a bunch of scripts in Databricks that perform a decent amount of data-wrangling. All of these scripts contain sensitive information and I have no intention of making them public.I would like to provide a service to my customers - so they ca...

  • 1103 Views
  • 4 replies
  • 0 kudos
Latest Reply
Atanu
Esteemed Contributor
  • 0 kudos

@Tamas D​  I understood your concern. For cluster creation in different subscription I think that's by design at this moment. But I would like to request you to add your use case to https://feedback.azure.com/d365community/forum/2efba7dc-ef24-ec11-b6...

  • 0 kudos
3 More Replies
Mohit_m
by Valued Contributor II
  • 339 Views
  • 1 replies
  • 2 kudos

docs.databricks.com

Are EBS volumes used by Databricks Clusters are encrypted especially the root volumes

  • 339 Views
  • 1 replies
  • 2 kudos
Latest Reply
Mohit_m
Valued Contributor II
  • 2 kudos

Yes these EBS volumes are encrypted. Earlier root volume encryptions were not supported but recently this encryption is also enabled (since Apr, 2021)please find more details on the below docs pagehttps://docs.databricks.com/clusters/configure.html#e...

  • 2 kudos
Anand_Ladda
by Honored Contributor II
  • 602 Views
  • 1 replies
  • 0 kudos
  • 602 Views
  • 1 replies
  • 0 kudos
Latest Reply
Ryan_Chynoweth
Honored Contributor III
  • 0 kudos

Generally, interactive clusters and jobs are better suited for data engineering and transformations as they support more than just SQL. However, if you are using pure SQL, then endpoints can be used for data transformations. All of the Spark SQL fun...

  • 0 kudos
Labels