https://community.databricks.com/t5/data-engineering/optimizing-a-batch-load-process-reading-with-the-jdbc-driver/m-p/21086#M14324 <P>I am doing a batch load, using the JDBC driver from a database table. I am noticing in Sparkui, that there is both memory and disk spill, but only on one executor. I am also, noticing that when trying to use the JDBC parallel read, it seems to run slower, then leaving it to default. </P><P></P><P>Some details:</P><P></P><UL><LI>I have 4 workers, 8 GB</LI><LI>The source table is around 80 million rows</LI><LI>I am using a "dateloaded" as the partition column.</LI><LI>&nbsp;sqlContext.setConf("spark.sql.shuffle.partitions","4"), set the shuffle partition size. Is it correct to set the shuffle the executor counts?</LI><LI>numPartitions=12, is it correct that it's ideal to have 3-4 task per executor?</LI></UL><P></P><P>The "dateloaded" is not a primary key, but is index. Is the spill a result of data skew? or have I set too few/many partitions for the shuffle or read? </P> Tue, 22 Nov 2022 22:47:12 GMT huyd 2022-11-22T22:47:12Z https://community.databricks.com/t5/data-engineering/optimizing-a-batch-load-process-reading-with-the-jdbc-driver/m-p/21086#M14324 <P>I am doing a batch load, using the JDBC driver from a database table. I am noticing in Sparkui, that there is both memory and disk spill, but only on one executor. I am also, noticing that when trying to use the JDBC parallel read, it seems to run slower, then leaving it to default. </P><P></P><P>Some details:</P><P></P><UL><LI>I have 4 workers, 8 GB</LI><LI>The source table is around 80 million rows</LI><LI>I am using a "dateloaded" as the partition column.</LI><LI>&nbsp;sqlContext.setConf("spark.sql.shuffle.partitions","4"), set the shuffle partition size. Is it correct to set the shuffle the executor counts?</LI><LI>numPartitions=12, is it correct that it's ideal to have 3-4 task per executor?</LI></UL><P></P><P>The "dateloaded" is not a primary key, but is index. Is the spill a result of data skew? or have I set too few/many partitions for the shuffle or read? </P> Tue, 22 Nov 2022 22:47:12 GMT https://community.databricks.com/t5/data-engineering/optimizing-a-batch-load-process-reading-with-the-jdbc-driver/m-p/21086#M14324 huyd 2022-11-22T22:47:12Z