cancel
Showing results for 
Search instead for 
Did you mean: 
Community Platform Discussions
Connect with fellow community members to discuss general topics related to the Databricks platform, industry trends, and best practices. Share experiences, ask questions, and foster collaboration within the community.
cancel
Showing results for 
Search instead for 
Did you mean: 

Forum Posts

alesventus
by Contributor
  • 996 Views
  • 0 replies
  • 0 kudos

Save dataframe to the same variable

I would like to know if there is any difference if I save dataframe during tranformation to itself as first code or to new dataframe as second example.Thankslog_df = log_df.withColumn("process_timestamp",from_utc_timestamp(lit(current_timestamp()),"E...

  • 996 Views
  • 0 replies
  • 0 kudos
Mohsen
by New Contributor
  • 1949 Views
  • 0 replies
  • 0 kudos

iceberg

Hi fellasi am working on databricks using icebergat first i have configured my notebook as belowspark.conf.set("spark.sql.catalog.spark_catalog","org.apache.iceberg.spark.SparkCatalog")spark.conf.set("spark.sql.catalog.spark_catalog.type", "hadoop")s...

  • 1949 Views
  • 0 replies
  • 0 kudos
olegmir
by New Contributor III
  • 1706 Views
  • 1 replies
  • 1 kudos

Resolved! threads leakage when getConnection fails

Hi,we are using databricks jdbc https://mvnrepository.com/artifact/com.databricks/databricks-jdbc/2.6.33it seems like there is a thread leakage when getConnection failscould anyone advice?can be reproduced with @Test void databricksThreads() {...

  • 1706 Views
  • 1 replies
  • 1 kudos
Latest Reply
olegmir
New Contributor III
  • 1 kudos

Hi,none of the above suggestion will not work...we already contacted databricks jdbc team, thread leakage was confirmed and was fixed in version 2.6.34https://mvnrepository.com/artifact/com.databricks/databricks-jdbc/2.6.34this leakage still exist if...

  • 1 kudos
Policepatil
by New Contributor III
  • 1025 Views
  • 0 replies
  • 0 kudos

Missing records while using limit in multithreading

Hi,I need to process nearly 30 files from different locations and insert records to RDS. I am using multi-threading to process these files parallelly like below. Test data:             I have configuration like below based on column 4: If column 4=0:...

image.png
  • 1025 Views
  • 0 replies
  • 0 kudos
priyakant1
by New Contributor II
  • 942 Views
  • 1 replies
  • 0 kudos

Suspension of Data Engineer Professional exam

Hi Databricks TeamI had scheduled my exam on 6th sep 2023, during exam same pop up came up, stating that I am looking in some other direction. I told them that my laptop mouse is not working properly, so I was looking at it. But still they suspended ...

  • 942 Views
  • 1 replies
  • 0 kudos
Latest Reply
sirishavemula20
New Contributor III
  • 0 kudos

Hi @priyakant1 ,Have you got any response from the team, like did they reschedule your exam?

  • 0 kudos
sirishavemula20
by New Contributor III
  • 2420 Views
  • 1 replies
  • 0 kudos

My exam has suspended , Need help Urgently (21/08/2023)

Hello Team,I encountered Pathetic experience while attempting my 1st DataBricks certification. Abruptly, Proctor asked me to show my desk, after showing he/she asked multiple times.. wasted my time and then suspended my exam.I want to file a complain...

  • 2420 Views
  • 1 replies
  • 0 kudos
Latest Reply
sirishavemula20
New Contributor III
  • 0 kudos

Sub: My exam Datbricks Data Engineer Associate got suspended_need immediate help please (10/09/2023)I encountered Pathetic experience while attempting my DataBricks Data engineer certification. Abruptly, Proctor asked me to show my desk, after showin...

  • 0 kudos
Policepatil
by New Contributor III
  • 2849 Views
  • 1 replies
  • 1 kudos

Resolved! Records are missing while filtering the dataframe in multithreading

 Hi, I need to process nearly 30 files from different locations and insert records to RDS. I am using multi-threading to process these files parallelly like below.   Test data:               I have configuration like below based on column 4: If colum...

Policepatil_0-1694077661899.png
  • 2849 Views
  • 1 replies
  • 1 kudos
Latest Reply
sean_owen
Databricks Employee
  • 1 kudos

Looks like you are comparing to strings like "1", not values like 1 in your filter condition. It's hard to say, there are some details missing like the rest of the code and the DF schema, and what output you are observing.

  • 1 kudos
Policepatil
by New Contributor III
  • 6097 Views
  • 0 replies
  • 0 kudos

Is it good to process files in multithreading?

Hi,I need to process nearly 30 files from different locations and insert records to RDS.I am using multi-threading to process these files parallelly like below. def process_files(file_path):    <process files here>    1. Find bad records based on fie...

  • 6097 Views
  • 0 replies
  • 0 kudos
bachan
by New Contributor II
  • 1639 Views
  • 1 replies
  • 0 kudos

Data Insertion

Scenario: Data from blob storage to SQL db once a week.I have 15(from current date to next 15 days) days data into the blob storage, stored date wise in parquet format, and after seven days the next 15 days data will be inserted. Means till 7th day t...

  • 1639 Views
  • 1 replies
  • 0 kudos
Sivaji
by New Contributor
  • 965 Views
  • 1 replies
  • 0 kudos

Databricks data engineer associate Exam got suspended.

Hello Team, I encountered Pathetic experience while attempting my 1st DataBricks certification. Abruptly, Proctor asked me to show my desk, after showing he/she asked multiple times.. wasted my time and then suspended my exam. I want to file a compla...

Community Platform Discussions
Data engineer Associate
Exam.
  • 965 Views
  • 1 replies
  • 0 kudos
Latest Reply
Cert-Team
Databricks Employee
  • 0 kudos

Hi @Sivaji Sorry to hear you had a bad experience, and that you got a slow response here in the community. I see that you have taken and passed the exam, Congratulations!For the future, our support team handles cases from here first so it tends to be...

  • 0 kudos
JRL
by New Contributor II
  • 1618 Views
  • 1 replies
  • 0 kudos

Github "Danger Zone"

There is a "Danger zone" appearing in the Github indicating that the repositories I share on Databricks should be Suspended and possibly that Databricks should be uninstalled.   This may be something standard in Github.  Has anyone run across it?

DangerZone.PNG
  • 1618 Views
  • 1 replies
  • 0 kudos
Latest Reply
sean_owen
Databricks Employee
  • 0 kudos

It's not telling you that you should do these things. It's telling you that you may break stuff by doing these things. Yes the "Danger Zone" is a thing on Github, it tries to warn you before you do things like click to delete a repo.

  • 0 kudos
jermaineharsh
by New Contributor III
  • 963 Views
  • 0 replies
  • 1 kudos

How to switch from free trial to Community Edition of Databricks in my Azure workspace?

hello,I am trying to switch into Databricks Community Edition after a 14 day trial. I was able to register, but when I try to start my new cluster, I get an error message, "Cluster start feature is currently disabled, and the cluster does not run".In...

  • 963 Views
  • 0 replies
  • 1 kudos