How nulls are stored in delta lake and databricks?

[DB_developer]

In my findings I have found a lot of delta tables in the lake house to be sparse so just wondering what space data lake takes to store null data and also any suggestions to handle sparse data tables in lake house would be appreciated.

I also want to optimize this sparse data at processing layer as well. We use databricks for our ETL operations. So, Can you also let me know how nulls are stored in databricks as well?

Thanks in advance!

[Ajay-Pandey]

Hi @Akash Ragothu​ please refer this link it might help you with that.

[Hubert-Dudek · Accepted solution]

As delta uses parquet files to store data inside delta:

"Nullity is encoded in the definition levels (which is run-length encoded). NULL values are not encoded in the data. For example, in a non-nested schema, a column with 1000 NULLs would be encoded with run-length encoding (0, 1000 times) for the definition levels and nothing else."

source: https://parquet.apache.org/docs/file-format/nulls/

[DB_developer]

That is useful info. Thanks! Can you also please let me know how many bytes of storage would a null value take in lakehouse?​

[Kaniz]

Hi @Akash Ragothu​, We haven’t heard from you since the last response from @Ajay Pandey​, and I was checking back to see if his suggestions helped you.

Or else, If you have any solution, please share it with the community, as it can be helpful to others.

Also, Please don't forget to click on the "Select As Best" button whenever the information provided helps resolve your question.