cancel
Showing results for 
Search instead for 
Did you mean: 
Community Discussions
Connect with fellow community members to discuss general topics related to the Databricks platform, industry trends, and best practices. Share experiences, ask questions, and foster collaboration within the community.
cancel
Showing results for 
Search instead for 
Did you mean: 

Forum Posts

div19882021
by New Contributor
  • 584 Views
  • 1 replies
  • 1 kudos

Is there a solution that we can display the worker types based on spark version selection using api?

Is there a solution that allows us to display the worker types or driver types based on the selection of Spark version using an api?

  • 584 Views
  • 1 replies
  • 1 kudos
Latest Reply
sean_owen
Honored Contributor II
  • 1 kudos

Can you clarify what you mean? Worker and driver types are not related to Spark version.

  • 1 kudos
pabloanzorenac
by New Contributor II
  • 1073 Views
  • 2 replies
  • 2 kudos

Resolved! Reduce EBS Default Volumes

By default Databricks creates 2 volumes: one with 30GB and the other one with 150GB. We have a lot of nodes in our pools and so a los of Terabytes of Volumes, but we are not making any use of them in the jobs. Is there any way to reduce the volumes? ...

  • 1073 Views
  • 2 replies
  • 2 kudos
Latest Reply
sean_owen
Honored Contributor II
  • 2 kudos

Yes, EBS vols are essential for shuffle spill for example. You are probably using them!

  • 2 kudos
1 More Replies
KrishZ
by Contributor
  • 3558 Views
  • 2 replies
  • 0 kudos

Uninstalling a preinstalled python package from Databricks

[Datasets](https://pypi.org/project/datasets/) python package comes preinstalled on databricks clusters. I want to uninstall it or completely prevent it's installation when I create/start a cluster.I couldn't find any solution on stackoverflow.And I ...

  • 3558 Views
  • 2 replies
  • 0 kudos
Latest Reply
sean_owen
Honored Contributor II
  • 0 kudos

@Kaniz_Fatma note that you can't actually uninstall packages in the runtime with pip.

  • 0 kudos
1 More Replies
FutureLegend
by New Contributor III
  • 1252 Views
  • 1 replies
  • 0 kudos

change cloud provider from AWS to GOOGLE

I registered a Databricks account and selected using AWS as cloud provider, may I know how to change it to using Google? Thanks!

  • 1252 Views
  • 1 replies
  • 0 kudos
Latest Reply
Kaniz_Fatma
Community Manager
  • 0 kudos

Hi @FutureLegend, To change the cloud provider from AWS to Google after registering a Databricks account, you must cancel your current Databricks subscription and sign up for a new subscription using Google Cloud Marketplace. Here are the steps to f...

  • 0 kudos
CraiMacl_23588
by New Contributor
  • 1029 Views
  • 1 replies
  • 0 kudos

Intermittent (cert) failure when connecting to AWS RDS

I've just upgraded a bunch of jobs to 12.2 LTS runtime and now getting intermittent failures with the following message:```java.sql.SQLException: [Amazon](600000) Error setting/closing connection: PKIX path building failed: sun.security.provider.cert...

  • 1029 Views
  • 1 replies
  • 0 kudos
Latest Reply
Kaniz_Fatma
Community Manager
  • 0 kudos

Hi @CraiMacl_23588, This error is related to an SSL certificate validation failure. To resolve the issue, you can try the following steps: • Check if the RDS instance SSL certificate is valid and not expired. • Ensure the root and intermediate certi...

  • 0 kudos
LiliL
by New Contributor
  • 879 Views
  • 1 replies
  • 2 kudos

create UDF in pyspark

Hi, Need the help of this community, unfortunately creating udfs is not my strongest skill set.I need to create UDF that will join two tables together, the problem is that one table has two id columns Name Table has id1 and id2 Transaction Table has ...

LiliL_0-1690464091302.png LiliL_1-1690464169973.png
  • 879 Views
  • 1 replies
  • 2 kudos
Latest Reply
Siebert_Looije
Contributor
  • 2 kudos

Hi,I am not sure if I understand your question directly but let me give it a try:- The constraint is if id2 in name table populated then join with id2: So I think you can also could first make a column called 'id' in which you get id2 if it is popula...

  • 2 kudos
Mahajan
by New Contributor II
  • 4400 Views
  • 6 replies
  • 2 kudos

Want to disable cell scrollers.

There are two scrollers visible in my notebook, 1 for cell and another is for notebook. How can i disable cell scroller sicne i am having a hard time to navigate to my code scrolling the cell every time.

Community Discussions
Notebook
scrollers
  • 4400 Views
  • 6 replies
  • 2 kudos
Latest Reply
UmaMahesh1
Honored Contributor III
  • 2 kudos

Hi @Mahajan What exactly do you mean by disabling the cell scroll ?  If and all there is an option as such, then it basically means you can't scroll the cell at all and the cell view is fixed. This makes the cell redundant as at any given point of ti...

  • 2 kudos
5 More Replies
DineshKumar
by New Contributor III
  • 2555 Views
  • 6 replies
  • 0 kudos

Databricks Cluster is going down after installing the external library

 I have created a Databricks cluster with below configurations.Databricks Runtime Version13.2 ML (includes Apache Spark 3.4.0, Scala 2.12)Node typei3.xlarge30.5 GB Memory, 4 CoresI created a notebook and trying to load the Mysql table which resides i...

  • 2555 Views
  • 6 replies
  • 0 kudos
Latest Reply
Debayan
Esteemed Contributor III
  • 0 kudos

Hi, The below error describes that there is an issue connecting to the host from Databricks, you can find more details about the network configurations here at https://docs.databricks.com/administration-guide/cloud-configurations/aws/customer-managed...

  • 0 kudos
5 More Replies
KoenZandvliet
by New Contributor III
  • 818 Views
  • 1 replies
  • 2 kudos

Resolved! Setting cluster permissions with API

I would like to update the permissions of a cluster using the API. Documentation mentions the following: patch api/2.0/permissions/{request_object_type}/{request_object_id}.Which {request_object_type} to use? ‘cluster’, ‘cluster’ and ‘compute’ are no...

  • 818 Views
  • 1 replies
  • 2 kudos
Latest Reply
daniel_sahal
Esteemed Contributor
  • 2 kudos

@KoenZandvliet clusters is the one you should be looking for.

  • 2 kudos
NewContributor
by New Contributor III
  • 2892 Views
  • 8 replies
  • 3 kudos

Resolved! Databricks Certified Data Engineer Associate (Version 2) Exam got suspended

Hi Team,My Databricks Certified Data Engineer Associate (Version 2) exam got suspended today and it is in suspended state.I was there continuously in front of the camera and suddenly the alert appeared and support person asked me to show the full tab...

  • 2892 Views
  • 8 replies
  • 3 kudos
Latest Reply
Rob_79
New Contributor II
  • 3 kudos

Hi @Kaniz_Fatma ,I've been into the same situation as Shifa and I've also raised ticket with Databricks but no feedback yet!Can you please help on that?Cheers,Rabie

  • 3 kudos
7 More Replies
nadishancosta
by New Contributor II
  • 725 Views
  • 2 replies
  • 0 kudos

Cannot access community account

Resetting password does not work. After I enter my new password, it just keeps processing. I waited for over 10 minutes, tried on different browsers, tried on a VPN, nothing works. Also this randomly happened. I didnt forget my password, just the sys...

  • 725 Views
  • 2 replies
  • 0 kudos
Latest Reply
nadishancosta
New Contributor II
  • 0 kudos

Its for the Community Edition

  • 0 kudos
1 More Replies
aupres
by New Contributor III
  • 1763 Views
  • 3 replies
  • 0 kudos

Resolved! How to generate schema with org.apache.spark.sql.functions.schema_of_csv?

Hello, I use spark 3.4.1-hadooop 3 on windows 11. And I am struggling to generate the schema of csv data with schema_of csv function. Below is my java codes. Map<String, String> kafkaParams = new HashMap<>(); kafkaParams.put("kafka.bootstrap.servers"...

Community Discussions
schema_of_csv
spark-java
  • 1763 Views
  • 3 replies
  • 0 kudos
Latest Reply
aupres
New Contributor III
  • 0 kudos

I use org.apache.spark.sql.functions.lit method and solve this issue. Thank you any way.

  • 0 kudos
2 More Replies
zyang
by Contributor
  • 6008 Views
  • 5 replies
  • 3 kudos

Sync the production data in environment into test environment

Hello,I have a database called sales which contain several delta tables and views in both production and test workspace. But the data is not synced because some people develop the code in test workspace. As time passed, both the data and the tables i...

  • 6008 Views
  • 5 replies
  • 3 kudos
Latest Reply
Kaniz_Fatma
Community Manager
  • 3 kudos

Hi @zyang,  To sync data and tables/views between production and test workspaces in Azure, the recommended approach is to use the Databricks Sync (DBSync) project, which is an object synchronization tool that backs up, restores, and syncs Databricks ...

  • 3 kudos
4 More Replies
Chris_Shehu
by Valued Contributor III
  • 503 Views
  • 0 replies
  • 0 kudos

Feature Request: GUI: Additional Collapse options

When you're using a very large notebook sometimes it gets frustrating scrolling through all the code blocks. It would be nice to have a few additional options to make this easier. 1) Add a collapse all code cells button to the top.2) Add a collapse a...

Community Discussions
Enhancement
Feature
GUI
Request
  • 503 Views
  • 0 replies
  • 0 kudos
Join 100K+ Data Experts: Register Now & Grow with Us!

Excited to expand your horizons with us? Click here to Register and begin your journey to success!

Already a member? Login and join your local regional user group! If there isn’t one near you, fill out this form and we’ll create one for you to join!