Databricks Community

vijaypodili · 12-20-2024

df_CorpBond= spark.read.format("parquet").option("header", "true").load(f"/mnt/{container_name}/raw_data/dsl.corporate.parquet") df_CorpBond.repartition(20).write\ .format("jdbc")\ .option("url", url_connector)\ .option("dbtable", "MarkIt...

vijaypodili · 12-29-2024

Hi @RiyazAliM this is my dag digramfile size is 3.5 gb and in future we need to load 14gb as well

vijaypodili · 12-24-2024

i removed the numpartiton,batch size and repartition as well job will take almost 3hrs to write data into ssms tables

vijaypodili · 12-23-2024

Hi @RiyazAliM i just want want to decrease the job time so that why i was using repartition,batch size,num partiton, but i did not work can you please suggest correct code this my worker node details 1-2 Workers32-64 GB Memory4-8 Cores1 Driver32 GB M...

vijaypodili · 12-21-2024

i'm trying to say my cluster have enough storage space 94gb so it can easy handle 2.3 gb data but my job taking longer time time job 2 and 3 completed with in the 3 min but job 4 taking longer to to complete its tasks

vijaypodili · 12-21-2024

i'm trying to say my cluster have enough storage space 94gb so it can easy handle 2.3 gb data but my job taking longer time time job 2 and 3 completed with in the 3 min but job 4 taking longer to to complete its tasks

Databricks Community

User Stats

User Activity

databricks job taking longer time to load 2.3 gb data from bolb to ssms table

Re: databricks job taking longer time to load 2.3 gb data from bolb to ssms table

Re: databricks job taking longer time to load 2.3 gb data from bolb to ssms table

Re: databricks job taking longer time to load 2.3 gb data from bolb to ssms table

Re: databricks job taking longer time to load 2.3 gb data from bolb to ssms table

Re: databricks job taking longer time to load 2.3 gb data from bolb to ssms table