Databricks Community

sshukla · 09-18-2024

Hi Team,I am reading 60 million -80million data from mysql server and writing into ADLS in parquet format but i am getting java heap issue, GC allocation failure and out of memory issue.below are my cluster configuration Driver - 56GB Ram, 16 coreWo...

sshukla · 09-19-2024

Hi @Witold Now i am able to read the data but one issue i am seeing is that out of 8 executor 3 are getting success in just 2-3 sec and rest 5 are running why this behavior.below is the coderemote_table1 = (spark.read .format("jdbc") .option("url",...

sshukla · 09-19-2024

Hi @Witold After trying table = (spark.read.format("jdbc").option("url", "<jdbc-url>").option("dbtable", "<table-name>").option("user", "<username>").option("password", "<password>").option("fetchSize", "1000") -- to 50000.load())job is taking lots o...

sshukla · 09-19-2024

thanks for the replylet me try the approaches which you mentioned and see the performance. will update you shortly.

sshukla · 09-19-2024

Hi,below is the codedriver=“com.mysql.cj. jdbc.Driver'database_host =“ip address"database_port=“3306"database_name =“database"table =“table"user ="user”password =“0password"url = f"jdbc:mysq]://(database_host): (database_port)/(database_name)?zeroDat...

sshukla · 09-18-2024

every time I am seeing data behaviour like this

Databricks Community

User Stats

User Activity

Java heap issue, GC allocation failure while writing data from mysql to adls

Re: Java heap issue, GC allocation failure while writing data from mysql to adls

Re: Java heap issue, GC allocation failure while writing data from mysql to adls

Re: Java heap issue, GC allocation failure while writing data from mysql to adls

Re: Java heap issue, GC allocation failure while writing data from mysql to adls

Re: Java heap issue, GC allocation failure while writing data from mysql to adls