cancel
Showing results for 
Search instead for 
Did you mean: 
Community Platform Discussions
Connect with fellow community members to discuss general topics related to the Databricks platform, industry trends, and best practices. Share experiences, ask questions, and foster collaboration within the community.
cancel
Showing results for 
Search instead for 
Did you mean: 

Forum Posts

GeKo
by New Contributor III
  • 933 Views
  • 0 replies
  • 0 kudos

global init script from workspace file ?

Hi Community,based on the announced change on Sep 1st, disabling cluster scoped init scripts in DBFS, I have questions re *global* init scripts.I am creating global init scripts via terraform "databricks_global_init_script" resources. Where do those ...

Community Platform Discussions
databricks_global_init_script
init script
workspace file
  • 933 Views
  • 0 replies
  • 0 kudos
shanmukh_b
by New Contributor
  • 17488 Views
  • 1 replies
  • 0 kudos

Convert string date to date after changing format

Hi,I am using Data bricks SQL and came across a scenario. I have a date field whose dates are in format of 'YYYY-MM-DD'. I changed their format into 'MM/DD/YYYY' using DATE_FORMAT() function.EFF_DT = 2000-01-14   EFF_DT _2 = DATE_FORMAT(EFF_DT, 'MM/d...

Community Platform Discussions
Databricks SQL
date
sql
string
  • 17488 Views
  • 1 replies
  • 0 kudos
Latest Reply
-werners-
Esteemed Contributor III
  • 0 kudos

if you use to_date, you will get a date column as mentioned above.If you want to use the format MM/dd/yyyy you can use date_format but this will return a string column.In order to use Spark date functions, Date string should comply with Spark DateTyp...

  • 0 kudos
DineshKumar
by New Contributor III
  • 1311 Views
  • 1 replies
  • 0 kudos

How to install AWS .pem file in databricks cluster to make a db connection to MySql RDS

I am trying to make a connection between AWS Mysql RDS and Databricks. I am using the below code to establish the connection. But its failed due to certificate is not installed. I have the .pem file with me. Could anyone help on how install this in D...

  • 1311 Views
  • 1 replies
  • 0 kudos
Latest Reply
Debayan
Databricks Employee
  • 0 kudos

Hi, Could you please provide the error code or the full error stack? Please tag @Debayan with your next comment which will notify me. Thank you!

  • 0 kudos
FutureLegend
by New Contributor III
  • 4106 Views
  • 2 replies
  • 1 kudos

Resolved! Download Dolly model on local machine

Hi~ I am new to LLM engineering, and am trying to download the Dolly-v2-7b model on local machine, so I don't need to connect to internet each time I am going to run the Dolly-v2-7b. Is it possible to do that? Thanks a lot!

  • 4106 Views
  • 2 replies
  • 1 kudos
Latest Reply
FutureLegend
New Contributor III
  • 1 kudos

Hi Kaniz and Sean, thanks for your responses and time.I was trying Kaniz's method, but got a reply from Sean, so I tried that too. I downloaded the file from the link Sean provided and saved it on my local machine, then used the code for Dollyv2 (htt...

  • 1 kudos
1 More Replies
TalY
by New Contributor II
  • 6779 Views
  • 5 replies
  • 0 kudos

Python notebook crashes with "The Python kernel is unresponsive"

While using a Python notebook that works on my machine it crashes on the same point with the errors "The Python kernel is unresponsive" and "The Python process exited with exit code 134 (SIGABRT: Aborted).",  but with no stacktrace for debugging the ...

  • 6779 Views
  • 5 replies
  • 0 kudos
Latest Reply
TalY
New Contributor II
  • 0 kudos

I am using the following DBR 12.2 LTS (includes Apache Spark 3.3.2, Scala 2.12).Fatal error: The Python kernel is unresponsive.--------------------------------------------------------------------------- The Python process exited with exit code 134 (S...

  • 0 kudos
4 More Replies
Hani4hanuman
by New Contributor II
  • 2009 Views
  • 2 replies
  • 1 kudos

Databricks notebook issue

Hi,I'm trying to run ADF pipeline.However, it is getting fail at Notebook activity with below error.Error :NoSuchMethodError: com.microsoft.sqlserver.jdbc.SQLServerBulkCopy.writeToServer(Lcom/microsoft/sqlserver/jdbc/ISQLServerBulkRecord;)V I think i...

  • 2009 Views
  • 2 replies
  • 1 kudos
Latest Reply
Hani4hanuman
New Contributor II
  • 1 kudos

@shan_chandra   Thanks for your reply as per your suggetion changed Databricks version from 9.1LTS to 12.2LTSBut after change this when i check library which you provided(i.e com.microsoft.azure:spark-mssql-connector_2.12:1.3.0) under Maven it is not...

  • 1 kudos
1 More Replies
lightningStrike
by New Contributor III
  • 2763 Views
  • 3 replies
  • 0 kudos

unable to install pymqi in azure databricks

Hi,I am trying to install pymqi via below command:pip install pymqi However, I am getting below error message:Python interpreter will be restarted. Collecting pymqi Using cached pymqi-1.12.10.tar.gz (91 kB) Installing build dependencies: started Inst...

lightningStrike_0-1688995466000.png
  • 2763 Views
  • 3 replies
  • 0 kudos
Latest Reply
sean_owen
Databricks Employee
  • 0 kudos

I don't think so, because it won't be specific to Databricks - this is all a property of the third party packages. And, there are billions of possible library conflicts. But this is not an example of a package conflict. It's an example of not complet...

  • 0 kudos
2 More Replies
alejandrofm
by Valued Contributor
  • 3923 Views
  • 1 replies
  • 1 kudos

Resolved! Configure job to use one cluster instance to multiple jobs

Hi! I have several tiny jobs that run in parallel and I want them to run on the same cluster:- Tasks type Python Script: I send the parameters this way to run the pyspark scripts.- Job compute cluster created as (copied JSON from Databricks Job UI)Ho...

Community Platform Discussions
cluster
job
job cluster
  • 3923 Views
  • 1 replies
  • 1 kudos
Latest Reply
KoenZandvliet
New Contributor III
  • 1 kudos

Unfortunately, running multiple jobs in parallel using a single job cluster is not supported (yet). New in databricks is the possibility to create a job that orchestrates multiple jobs. These jobs will however still use their own cluster (configurati...

  • 1 kudos
div19882021
by New Contributor
  • 867 Views
  • 1 replies
  • 1 kudos

Is there a solution that we can display the worker types based on spark version selection using api?

Is there a solution that allows us to display the worker types or driver types based on the selection of Spark version using an api?

  • 867 Views
  • 1 replies
  • 1 kudos
Latest Reply
sean_owen
Databricks Employee
  • 1 kudos

Can you clarify what you mean? Worker and driver types are not related to Spark version.

  • 1 kudos
pabloanzorenac
by New Contributor II
  • 1833 Views
  • 2 replies
  • 2 kudos

Resolved! Reduce EBS Default Volumes

By default Databricks creates 2 volumes: one with 30GB and the other one with 150GB. We have a lot of nodes in our pools and so a los of Terabytes of Volumes, but we are not making any use of them in the jobs. Is there any way to reduce the volumes? ...

  • 1833 Views
  • 2 replies
  • 2 kudos
Latest Reply
sean_owen
Databricks Employee
  • 2 kudos

Yes, EBS vols are essential for shuffle spill for example. You are probably using them!

  • 2 kudos
1 More Replies
KrishZ
by Contributor
  • 5293 Views
  • 1 replies
  • 0 kudos

Uninstalling a preinstalled python package from Databricks

[Datasets](https://pypi.org/project/datasets/) python package comes preinstalled on databricks clusters. I want to uninstall it or completely prevent it's installation when I create/start a cluster.I couldn't find any solution on stackoverflow.And I ...

  • 5293 Views
  • 1 replies
  • 0 kudos
Latest Reply
sean_owen
Databricks Employee
  • 0 kudos

@Retired_mod note that you can't actually uninstall packages in the runtime with pip.

  • 0 kudos
Chris_Shehu
by Valued Contributor III
  • 1698 Views
  • 0 replies
  • 0 kudos

Databricks Assistant HIPPA? Future Cost?

With the Public Preview of Databricks Assistant, I have a few questions. 1) If the Azure Tenet is HIPPA compliant does that compliance also include the Databricks Assistant features? 2) Right now the product is free but what will the cost be? Will we...

  • 1698 Views
  • 0 replies
  • 0 kudos
LiliL
by New Contributor
  • 1356 Views
  • 1 replies
  • 2 kudos

create UDF in pyspark

Hi, Need the help of this community, unfortunately creating udfs is not my strongest skill set.I need to create UDF that will join two tables together, the problem is that one table has two id columns Name Table has id1 and id2 Transaction Table has ...

LiliL_0-1690464091302.png LiliL_1-1690464169973.png
  • 1356 Views
  • 1 replies
  • 2 kudos
Latest Reply
Siebert_Looije
Contributor
  • 2 kudos

Hi,I am not sure if I understand your question directly but let me give it a try:- The constraint is if id2 in name table populated then join with id2: So I think you can also could first make a column called 'id' in which you get id2 if it is popula...

  • 2 kudos

Connect with Databricks Users in Your Area

Join a Regional User Group to connect with local Databricks users. Events will be happening in your city, and you won’t want to miss the chance to attend and share knowledge.

If there isn’t a group near you, start one and help create a community that brings people together.

Request a New Group
Top Kudoed Authors