https://community.databricks.com/t5/warehousing-analytics/unable-to-obtain-count-of-dataframe/m-p/91913#M1582 <P>I am unable to obtain a count of a dataframe, it always get stuck at 1 stage, I have tried reducing the size, what can be the issue? How can I read cluster logs to identify the issue?</P><DIV class=""><DIV class=""><DIV class=""><DIV class=""><DIV class=""><BR /><BR /><P>&nbsp;</P></DIV></DIV></DIV></DIV></DIV> Thu, 26 Sep 2024 15:38:40 GMT qwerty3 2024-09-26T15:38:40Z https://community.databricks.com/t5/warehousing-analytics/unable-to-obtain-count-of-dataframe/m-p/91913#M1582 <P>I am unable to obtain a count of a dataframe, it always get stuck at 1 stage, I have tried reducing the size, what can be the issue? How can I read cluster logs to identify the issue?</P><DIV class=""><DIV class=""><DIV class=""><DIV class=""><DIV class=""><BR /><BR /><P>&nbsp;</P></DIV></DIV></DIV></DIV></DIV> Thu, 26 Sep 2024 15:38:40 GMT https://community.databricks.com/t5/warehousing-analytics/unable-to-obtain-count-of-dataframe/m-p/91913#M1582 qwerty3 2024-09-26T15:38:40Z https://community.databricks.com/t5/warehousing-analytics/unable-to-obtain-count-of-dataframe/m-p/91918#M1584 <P><span class="lia-inline-image-display-wrapper lia-image-align-inline" image-alt="qwerty3_0-1727365156978.png" style="width: 400px;"><img src="https://community.databricks.com/t5/image/serverpage/image-id/11528iCB43B15648041CCB/image-size/medium?v=v2&amp;px=400" role="button" title="qwerty3_0-1727365156978.png" alt="qwerty3_0-1727365156978.png" /></span></P><P>&nbsp;</P> Thu, 26 Sep 2024 15:40:20 GMT https://community.databricks.com/t5/warehousing-analytics/unable-to-obtain-count-of-dataframe/m-p/91918#M1584 qwerty3 2024-09-26T15:40:20Z https://community.databricks.com/t5/warehousing-analytics/unable-to-obtain-count-of-dataframe/m-p/91920#M1585 <P>You should look at Spark Web UI and investigate information there.There could be several causes of this like:</P><P>- skewed data</P><P>- memory issues</P><P>- shuffling data</P><P>&nbsp;</P> Thu, 26 Sep 2024 15:42:52 GMT https://community.databricks.com/t5/warehousing-analytics/unable-to-obtain-count-of-dataframe/m-p/91920#M1585 szymon_dybczak 2024-09-26T15:42:52Z https://community.databricks.com/t5/warehousing-analytics/unable-to-obtain-count-of-dataframe/m-p/91926#M1588 <P>Driver memory is good enough, it is able to handle 90 lakhs data, what I am giving it is definitely less than that, what can I do about skewed data and shuffling?</P> Thu, 26 Sep 2024 15:50:54 GMT https://community.databricks.com/t5/warehousing-analytics/unable-to-obtain-count-of-dataframe/m-p/91926#M1588 qwerty3 2024-09-26T15:50:54Z