https://community.databricks.com/t5/data-engineering/how-to-optimize-storage-for-sparse-data-in-data-lake/m-p/17877#M11800 <P>datalake itself not, but the file format you use to store data does.</P><P>f.e. parquet uses column compression, so sparse data will compress pretty good.</P><P>csv on the other hand: total disaster</P> Thu, 08 Dec 2022 14:17:23 GMT -werners- 2022-12-08T14:17:23Z https://community.databricks.com/t5/data-engineering/how-to-optimize-storage-for-sparse-data-in-data-lake/m-p/17876#M11799 <P>I have lot of tables with 80% of columns being filled with nulls. I understand SQL sever provides a way to handle these kind of data during the data definition of the tables (with Sparse keyword). Do datalake provide similar kind of thing?</P><P></P> Thu, 08 Dec 2022 13:08:20 GMT https://community.databricks.com/t5/data-engineering/how-to-optimize-storage-for-sparse-data-in-data-lake/m-p/17876#M11799 DB_developer 2022-12-08T13:08:20Z https://community.databricks.com/t5/data-engineering/how-to-optimize-storage-for-sparse-data-in-data-lake/m-p/17877#M11800 <P>datalake itself not, but the file format you use to store data does.</P><P>f.e. parquet uses column compression, so sparse data will compress pretty good.</P><P>csv on the other hand: total disaster</P> Thu, 08 Dec 2022 14:17:23 GMT https://community.databricks.com/t5/data-engineering/how-to-optimize-storage-for-sparse-data-in-data-lake/m-p/17877#M11800 -werners- 2022-12-08T14:17:23Z https://community.databricks.com/t5/data-engineering/how-to-optimize-storage-for-sparse-data-in-data-lake/m-p/17878#M11801 <P>Unless you compress the entire CSV, which also should be a viable approach.</P><P></P><P>That said, Delta/Parquet would normally be the better option where each column in compressed.</P> Mon, 12 Dec 2022 08:15:26 GMT https://community.databricks.com/t5/data-engineering/how-to-optimize-storage-for-sparse-data-in-data-lake/m-p/17878#M11801 Håkon_Åmdal 2022-12-12T08:15:26Z