cancel
Turn on suggestions
Auto-suggest helps you quickly narrow down your search results by suggesting possible matches as you type.
Showing results for 
Search instead for 
Did you mean: 
Help
Community Discussions
Connect with fellow community members to discuss general topics related to the Databricks platform, industry trends, and best practices. Share experiences, ask questions, and foster collaboration within the community.
cancel
Turn on suggestions
Auto-suggest helps you quickly narrow down your search results by suggesting possible matches as you type.
Showing results for 
Search instead for 
Did you mean: 

Forum Posts

Start a conversation
DineshKumar
by • New Contributor III
  • 779 Views
  • 1 replies
  • 0 kudos

How to install AWS .pem file in databricks cluster to make a db connection to MySql RDS

I am trying to make a connection between AWS Mysql RDS and Databricks. I am using the below code to establish the connection. But its failed due to certificate is not installed. I have the .pem file with me. Could anyone help on how install this in D...

Community Discussions
Reply
  • 779 Views
  • 1 replies
  • 0 kudos
View Replies
Latest Reply
Debayan
Esteemed Contributor III
  • 0 kudos

Hi, Could you please provide the error code or the full error stack? Please tag @Debayan with your next comment which will notify me. Thank you!

  • 0 kudos
FutureLegend
by • New Contributor III
  • 2694 Views
  • 3 replies
  • 2 kudos

Resolved! Download Dolly model on local machine

Hi~ I am new to LLM engineering, and am trying to download the Dolly-v2-7b model on local machine, so I don't need to connect to internet each time I am going to run the Dolly-v2-7b. Is it possible to do that? Thanks a lot!

Community Discussions
Reply
  • 2694 Views
  • 3 replies
  • 2 kudos
View Replies
Latest Reply
FutureLegend
New Contributor III
  • 2 kudos

Hi Kaniz and Sean, thanks for your responses and time.I was trying Kaniz's method, but got a reply from Sean, so I tried that too. I downloaded the file from the link Sean provided and saved it on my local machine, then used the code for Dollyv2 (htt...

  • 2 kudos
2 More Replies
TalY
by • New Contributor II
  • 3904 Views
  • 7 replies
  • 0 kudos

Python notebook crashes with "The Python kernel is unresponsive"

While using a Python notebook that works on my machine it crashes on the same point with the errors "The Python kernel is unresponsive" and "The Python process exited with exit code 134 (SIGABRT: Aborted).",  but with no stacktrace for debugging the ...

Community Discussions
Reply
  • 3904 Views
  • 7 replies
  • 0 kudos
View Replies
Latest Reply
TalY
New Contributor II
  • 0 kudos

I am using the following DBR 12.2 LTS (includes Apache Spark 3.3.2, Scala 2.12).Fatal error: The Python kernel is unresponsive.--------------------------------------------------------------------------- The Python process exited with exit code 134 (S...

  • 0 kudos
6 More Replies
Hani4hanuman
by • New Contributor II
  • 1615 Views
  • 2 replies
  • 1 kudos

Databricks notebook issue

Hi,I'm trying to run ADF pipeline.However, it is getting fail at Notebook activity with below error.Error :NoSuchMethodError: com.microsoft.sqlserver.jdbc.SQLServerBulkCopy.writeToServer(Lcom/microsoft/sqlserver/jdbc/ISQLServerBulkRecord;)V I think i...

Community Discussions
Reply
  • 1615 Views
  • 2 replies
  • 1 kudos
View Replies
Latest Reply
Hani4hanuman
New Contributor II
  • 1 kudos

@shan_chandra   Thanks for your reply as per your suggetion changed Databricks version from 9.1LTS to 12.2LTSBut after change this when i check library which you provided(i.e com.microsoft.azure:spark-mssql-connector_2.12:1.3.0) under Maven it is not...

  • 1 kudos
1 More Replies
lightningStrike
by • New Contributor III
  • 1972 Views
  • 3 replies
  • 0 kudos

unable to install pymqi in azure databricks

Hi,I am trying to install pymqi via below command:pip install pymqi However, I am getting below error message:Python interpreter will be restarted. Collecting pymqi Using cached pymqi-1.12.10.tar.gz (91 kB) Installing build dependencies: started Inst...

lightningStrike_0-1688995466000.png
Community Discussions
Reply
  • 1972 Views
  • 3 replies
  • 0 kudos
View Replies
Latest Reply
sean_owen
Honored Contributor II
  • 0 kudos

I don't think so, because it won't be specific to Databricks - this is all a property of the third party packages. And, there are billions of possible library conflicts. But this is not an example of a package conflict. It's an example of not complet...

  • 0 kudos
2 More Replies
alejandrofm
by • Valued Contributor
  • 2652 Views
  • 1 replies
  • 1 kudos

Resolved! Configure job to use one cluster instance to multiple jobs

Hi! I have several tiny jobs that run in parallel and I want them to run on the same cluster:- Tasks type Python Script: I send the parameters this way to run the pyspark scripts.- Job compute cluster created as (copied JSON from Databricks Job UI)Ho...

Community Discussions
cluster
job
job cluster
Reply
  • 2652 Views
  • 1 replies
  • 1 kudos
View Replies
Latest Reply
KoenZandvliet
New Contributor III
  • 1 kudos

Unfortunately, running multiple jobs in parallel using a single job cluster is not supported (yet). New in databricks is the possibility to create a job that orchestrates multiple jobs. These jobs will however still use their own cluster (configurati...

  • 1 kudos
div19882021
by • New Contributor
  • 551 Views
  • 1 replies
  • 1 kudos

Is there a solution that we can display the worker types based on spark version selection using api?

Is there a solution that allows us to display the worker types or driver types based on the selection of Spark version using an api?

Community Discussions
Reply
  • 551 Views
  • 1 replies
  • 1 kudos
View Replies
Latest Reply
sean_owen
Honored Contributor II
  • 1 kudos

Can you clarify what you mean? Worker and driver types are not related to Spark version.

  • 1 kudos
pabloanzorenac
by • New Contributor II
  • 1019 Views
  • 2 replies
  • 2 kudos

Resolved! Reduce EBS Default Volumes

By default Databricks creates 2 volumes: one with 30GB and the other one with 150GB. We have a lot of nodes in our pools and so a los of Terabytes of Volumes, but we are not making any use of them in the jobs. Is there any way to reduce the volumes? ...

Community Discussions
Reply
  • 1019 Views
  • 2 replies
  • 2 kudos
View Replies
Latest Reply
sean_owen
Honored Contributor II
  • 2 kudos

Yes, EBS vols are essential for shuffle spill for example. You are probably using them!

  • 2 kudos
1 More Replies
KrishZ
by • Contributor
  • 3394 Views
  • 2 replies
  • 0 kudos

Uninstalling a preinstalled python package from Databricks

[Datasets](https://pypi.org/project/datasets/) python package comes preinstalled on databricks clusters. I want to uninstall it or completely prevent it's installation when I create/start a cluster.I couldn't find any solution on stackoverflow.And I ...

Community Discussions
Reply
  • 3394 Views
  • 2 replies
  • 0 kudos
View Replies
Latest Reply
sean_owen
Honored Contributor II
  • 0 kudos

@Kaniz_Fatma note that you can't actually uninstall packages in the runtime with pip.

  • 0 kudos
1 More Replies
FutureLegend
by • New Contributor III
  • 1197 Views
  • 1 replies
  • 0 kudos

change cloud provider from AWS to GOOGLE

I registered a Databricks account and selected using AWS as cloud provider, may I know how to change it to using Google? Thanks!

Community Discussions
Reply
  • 1197 Views
  • 1 replies
  • 0 kudos
View Replies
Latest Reply
Kaniz_Fatma
Community Manager
  • 0 kudos

Hi @FutureLegend, To change the cloud provider from AWS to Google after registering a Databricks account, you must cancel your current Databricks subscription and sign up for a new subscription using Google Cloud Marketplace. Here are the steps to f...

  • 0 kudos
CraiMacl_23588
by • New Contributor
  • 983 Views
  • 1 replies
  • 0 kudos

Intermittent (cert) failure when connecting to AWS RDS

I've just upgraded a bunch of jobs to 12.2 LTS runtime and now getting intermittent failures with the following message:```java.sql.SQLException: [Amazon](600000) Error setting/closing connection: PKIX path building failed: sun.security.provider.cert...

Community Discussions
Reply
  • 983 Views
  • 1 replies
  • 0 kudos
View Replies
Latest Reply
Kaniz_Fatma
Community Manager
  • 0 kudos

Hi @CraiMacl_23588, This error is related to an SSL certificate validation failure. To resolve the issue, you can try the following steps: • Check if the RDS instance SSL certificate is valid and not expired. • Ensure the root and intermediate certi...

  • 0 kudos
LiliL
by • New Contributor
  • 838 Views
  • 1 replies
  • 2 kudos

create UDF in pyspark

Hi, Need the help of this community, unfortunately creating udfs is not my strongest skill set.I need to create UDF that will join two tables together, the problem is that one table has two id columns Name Table has id1 and id2 Transaction Table has ...

LiliL_0-1690464091302.png LiliL_1-1690464169973.png
Community Discussions
pyspark
udf
Reply
  • 838 Views
  • 1 replies
  • 2 kudos
View Replies
Latest Reply
Siebert_Looije
Contributor
  • 2 kudos

Hi,I am not sure if I understand your question directly but let me give it a try:- The constraint is if id2 in name table populated then join with id2: So I think you can also could first make a column called 'id' in which you get id2 if it is popula...

  • 2 kudos
Mahajan
by • New Contributor II
  • 4183 Views
  • 6 replies
  • 2 kudos

Want to disable cell scrollers.

There are two scrollers visible in my notebook, 1 for cell and another is for notebook. How can i disable cell scroller sicne i am having a hard time to navigate to my code scrolling the cell every time.

Community Discussions
Notebook
scrollers
Reply
  • 4183 Views
  • 6 replies
  • 2 kudos
View Replies
Latest Reply
UmaMahesh1
Honored Contributor III
  • 2 kudos

Hi @Mahajan What exactly do you mean by disabling the cell scroll ?  If and all there is an option as such, then it basically means you can't scroll the cell at all and the cell view is fixed. This makes the cell redundant as at any given point of ti...

  • 2 kudos
5 More Replies
DineshKumar
by • New Contributor III
  • 2458 Views
  • 6 replies
  • 0 kudos

Databricks Cluster is going down after installing the external library

 I have created a Databricks cluster with below configurations.Databricks Runtime Version13.2 ML (includes Apache Spark 3.4.0, Scala 2.12)Node typei3.xlarge30.5 GB Memory, 4 CoresI created a notebook and trying to load the Mysql table which resides i...

Community Discussions
Reply
  • 2458 Views
  • 6 replies
  • 0 kudos
View Replies
Latest Reply
Debayan
Esteemed Contributor III
  • 0 kudos

Hi, The below error describes that there is an issue connecting to the host from Databricks, you can find more details about the network configurations here at https://docs.databricks.com/administration-guide/cloud-configurations/aws/customer-managed...

  • 0 kudos
5 More Replies
KoenZandvliet
by • New Contributor III
  • 774 Views
  • 1 replies
  • 2 kudos

Resolved! Setting cluster permissions with API

I would like to update the permissions of a cluster using the API. Documentation mentions the following: patch api/2.0/permissions/{request_object_type}/{request_object_id}.Which {request_object_type} to use? ‘cluster’, ‘cluster’ and ‘compute’ are no...

Community Discussions
Reply
  • 774 Views
  • 1 replies
  • 2 kudos
View Replies
Latest Reply
daniel_sahal
Esteemed Contributor
  • 2 kudos

@KoenZandvliet clusters is the one you should be looking for.

  • 2 kudos
Join 100K+ Data Experts: Register Now & Grow with Us!

Excited to expand your horizons with us? Click here to Register and begin your journey to success!

Already a member? Login and join your local regional user group! If there isn’t one near you, fill out this form and we’ll create one for you to join!

Labels