Databricks Job is slower.

Raymond_Garcia — Fri, 24 Jun 2022 18:18:52 GMT

Hello, I have a data bricks question. A Dataframe job that writes in an s3 bucket usually takes 8 minutes to finish, but now it takes from 8 to 9 hours to complete. Does anybody have some clues about this behavior?

the data frame size is about 300 or 400 records

it is a simple query in a delta table:

val results = spark
.table("table")
.filter()
.filter(by_date)
.drop(some_columns")
.select(a_struct_field)
.withColumn("image", image) 
 
listofString.foreach { mystring =>
  println(s"start writing .json to S3 for ${results}")
  results
  .filter($"struct.field.result" === results)
  .coalesce(1)
  .write
  .mode(SaveMode.Overwrite)
  .json(s"${filePath}/temp_${results}")
  println(s"complete writing .json to S3 for ${results}")
}

Thanks in advance

Re: Databricks Job is slower.

Raymond_Garcia — Tue, 28 Jun 2022 21:35:20 GMT

Hi thanks, I will check them out, and I will let you know. 🙂

Re: Databricks Job is slower.

Raymond_Garcia — Wed, 29 Jun 2022 15:27:18 GMT

Hello, I was able to reduce the time significantly. I used the OPTIMIZE keyword before starting processing.

Thanks!

topic Databricks Job is slower. in Data Engineering

Databricks Job is slower.

Re: Databricks Job is slower.

Re: Databricks Job is slower.