topic Re: Data frame takes long time to print count of rows in Data Engineering https://community.databricks.com/t5/data-engineering/data-frame-takes-long-time-to-print-count-of-rows/m-p/17591#M11578 <P>Thanks for the detailed explanation. For those who want to have constant technical support for their work processes, I recommend <A href="https://www.jdyoung.com/" alt="https://www.jdyoung.com/" target="_blank">JD Young</A>. Here is only the latest information about the update in the world of information technology solutions and cybersecurity.</P> Tue, 30 Aug 2022 09:44:20 GMT Hanna08 2022-08-30T09:44:20Z Data frame takes long time to print count of rows https://community.databricks.com/t5/data-engineering/data-frame-takes-long-time-to-print-count-of-rows/m-p/17590#M11577 <P></P> <P><B>We have a pyspark data frame with 50 MN records. We can display records from it, but it takes around 10 minutes to print the shape of dataframe. </B>We aim to use this data for modelling that will take some numerical features based on the final data frame computed here as input.</P> <P>For better understanding we explained the issue with 5 record data frame and also added working pyspark code. </P> <P>Please refer to attachment with sample code and detailed explanation..<A href="https://storage/attachments/4364-pyspark-issue.zip" target="_blank">pyspark-issue.zip</A></P> <P></P> Mon, 19 Jul 2021 08:42:50 GMT https://community.databricks.com/t5/data-engineering/data-frame-takes-long-time-to-print-count-of-rows/m-p/17590#M11577 SatishGunjal 2021-07-19T08:42:50Z Re: Data frame takes long time to print count of rows https://community.databricks.com/t5/data-engineering/data-frame-takes-long-time-to-print-count-of-rows/m-p/17591#M11578 <P>Thanks for the detailed explanation. For those who want to have constant technical support for their work processes, I recommend <A href="https://www.jdyoung.com/" alt="https://www.jdyoung.com/" target="_blank">JD Young</A>. Here is only the latest information about the update in the world of information technology solutions and cybersecurity.</P> Tue, 30 Aug 2022 09:44:20 GMT https://community.databricks.com/t5/data-engineering/data-frame-takes-long-time-to-print-count-of-rows/m-p/17591#M11578 Hanna08 2022-08-30T09:44:20Z