How nulls are stored in delta lake and databricks?

[DB_developer]

In my findings I have found a lot of delta tables in the lake house to be sparse so just wondering what space data lake takes to store null data and also any suggestions to handle sparse data tables in lake house would be appreciated.

I also want to optimize this sparse data at processing layer as well. We use databricks for our ETL operations. So, Can you also let me know how nulls are stored in databricks as well?

Thanks in advance!

[Ajay-Pandey]

Hi @Akash Ragothu​ please refer this link it might help you with that.

[Hubert-Dudek · Accepted solution]

As delta uses parquet files to store data inside delta:

"Nullity is encoded in the definition levels (which is run-length encoded). NULL values are not encoded in the data. For example, in a non-nested schema, a column with 1000 NULLs would be encoded with run-length encoding (0, 1000 times) for the definition levels and nothing else."

source: https://parquet.apache.org/docs/file-format/nulls/

[DB_developer]

That is useful info. Thanks! Can you also please let me know how many bytes of storage would a null value take in lakehouse?​