How to optimize storage for sparse data in data lake?

DB_developer
New Contributor III

‎12-08-2022 05:08 AM

I have lot of tables with 80% of columns being filled with nulls. I understand SQL sever provides a way to handle these kind of data during the data definition of the tables (with Sparse keyword). Do datalake provide similar kind of thing?

Labels:

-werners-
Esteemed Contributor III

‎12-08-2022 06:17 AM

datalake itself not, but the file format you use to store data does.

f.e. parquet uses column compression, so sparse data will compress pretty good.

csv on the other hand: total disaster

Håkon_Åmdal
New Contributor III
In response to -werners-

‎12-12-2022 12:15 AM

Unless you compress the entire CSV, which also should be a viable approach.

That said, Delta/Parquet would normally be the better option where each column in compressed.