topic Re: How nulls are stored in delta lake and databricks? in Data Engineering

How nulls are stored in delta lake and databricks?

DB_developer — Wed, 07 Dec 2022 09:27:39 GMT

In my findings I have found a lot of delta tables in the lake house to be sparse so just wondering what space data lake takes to store null data and also any suggestions to handle sparse data tables in lake house would be appreciated.

I also want to optimize this sparse data at processing layer as well. We use databricks for our ETL operations. So, Can you also let me know how nulls are stored in databricks as well?

Thanks in advance!

Re: How nulls are stored in delta lake and databricks?

Ajay-Pandey — Wed, 07 Dec 2022 10:59:53 GMT

Hi @Akash Ragothu please refer this link it might help you with that.

Re: How nulls are stored in delta lake and databricks?

Hubert-Dudek — Wed, 07 Dec 2022 11:38:52 GMT

As delta uses parquet files to store data inside delta:

"Nullity is encoded in the definition levels (which is run-length encoded). NULL values are not encoded in the data. For example, in a non-nested schema, a column with 1000 NULLs would be encoded with run-length encoding (0, 1000 times) for the definition levels and nothing else."

source: https://parquet.apache.org/docs/file-format/nulls/

Re: How nulls are stored in delta lake and databricks?

DB_developer — Wed, 07 Dec 2022 11:57:58 GMT

That is useful info. Thanks! Can you also please let me know how many bytes of storage would a null value take in lakehouse?