https://community.databricks.com/t5/get-started-discussions/big-time-differences-in-reading-tables/m-p/38380#M5501 <P>When I read managed table in #databricks# i can see big differences in time spent. Small test table with just 2 records is once loaded in 3 seconds and another time in 30 seconds. Reading table_change for this tinny table took 15 minutes. Don't know if this has any explanation or cluster setting is somehow wrong.</P><OL><LI>storage: ADLS gen2</LI><LI>cluster config:<UL><LI>access mode: shared</LI><LI><SPAN>runtime: 13.2 (includes Apache Spark 3.4.0, Scala 2.12)</SPAN></LI><LI><SPAN>worker: Standard_DS_v2 14GB, 4 Cores min 2 max 8</SPAN></LI><LI><SPAN>driver:&nbsp;Standard_DS_v2 </SPAN></LI></UL></LI><LI><SPAN>enable autoscaling = true</SPAN></LI><LI><SPAN>No other advanced options</SPAN></LI></OL><P><SPAN>Thanks.</SPAN></P><P>&nbsp;</P><P>&nbsp;</P> Tue, 25 Jul 2023 13:06:16 GMT alesventus 2023-07-25T13:06:16Z https://community.databricks.com/t5/get-started-discussions/big-time-differences-in-reading-tables/m-p/38380#M5501 <P>When I read managed table in #databricks# i can see big differences in time spent. Small test table with just 2 records is once loaded in 3 seconds and another time in 30 seconds. Reading table_change for this tinny table took 15 minutes. Don't know if this has any explanation or cluster setting is somehow wrong.</P><OL><LI>storage: ADLS gen2</LI><LI>cluster config:<UL><LI>access mode: shared</LI><LI><SPAN>runtime: 13.2 (includes Apache Spark 3.4.0, Scala 2.12)</SPAN></LI><LI><SPAN>worker: Standard_DS_v2 14GB, 4 Cores min 2 max 8</SPAN></LI><LI><SPAN>driver:&nbsp;Standard_DS_v2 </SPAN></LI></UL></LI><LI><SPAN>enable autoscaling = true</SPAN></LI><LI><SPAN>No other advanced options</SPAN></LI></OL><P><SPAN>Thanks.</SPAN></P><P>&nbsp;</P><P>&nbsp;</P> Tue, 25 Jul 2023 13:06:16 GMT https://community.databricks.com/t5/get-started-discussions/big-time-differences-in-reading-tables/m-p/38380#M5501 alesventus 2023-07-25T13:06:16Z