https://community.databricks.com/t5/data-engineering/get-size-of-a-column-in-bytes-for-a-pyspark-data-frame/m-p/27195#M19075 <P></P> <P>There isn't one size for a column; it takes some amount of bytes in memory, but a different amount potentially when serialized on disk or stored in Parquet. You can work out the size in memory from its data type; an array of 100 bytes takes 100 bytes; a long takes 8 bytes, etc.</P> <P></P> Fri, 17 Apr 2020 21:16:43 GMT sean_owen 2020-04-17T21:16:43Z https://community.databricks.com/t5/data-engineering/get-size-of-a-column-in-bytes-for-a-pyspark-data-frame/m-p/27194#M19074 <P></P> <P>Hello All,</P> <P></P><P></P>I have a column in a dataframe which i struct type.I want to find the size of the column in bytes.it is getting failed while loading in snowflake.<P></P>I could see size functions avialable to get the length.how to calculate the size in bytes for a column in pyspark dataframe.<P></P><P></P>pyspark.sql.functions.size(col)<P></P>Collection function: returns the length of the array or map stored in the column.<P></P><P></P>Please help me on this case.<P></P><P></P>Thanks <P></P> Thu, 16 Apr 2020 18:49:21 GMT https://community.databricks.com/t5/data-engineering/get-size-of-a-column-in-bytes-for-a-pyspark-data-frame/m-p/27194#M19074 Anbazhagananbut 2020-04-16T18:49:21Z https://community.databricks.com/t5/data-engineering/get-size-of-a-column-in-bytes-for-a-pyspark-data-frame/m-p/27195#M19075 <P></P> <P>There isn't one size for a column; it takes some amount of bytes in memory, but a different amount potentially when serialized on disk or stored in Parquet. You can work out the size in memory from its data type; an array of 100 bytes takes 100 bytes; a long takes 8 bytes, etc.</P> <P></P> Fri, 17 Apr 2020 21:16:43 GMT https://community.databricks.com/t5/data-engineering/get-size-of-a-column-in-bytes-for-a-pyspark-data-frame/m-p/27195#M19075 sean_owen 2020-04-17T21:16:43Z