topic Re: delta table storage in Data Engineering

delta table storage

Braxx — Fri, 27 May 2022 09:32:31 GMT

I couldn't find it clearly explained anywhere, so hope sb here shed some light on that.

Few questions:

1) Where does delta tables are stored?

Docs say: "Delta Lake uses versioned Parquet files to store your data in your cloud storage"

So where exactly is it stored? Could it be stored on any storage I use for instance blob storage or is it somewhere on DBFS or databricks cluster?

2) If I have already data saved to parquet on my Azure blob storage and want to convert them to delta, would this change be implemented on blob? Or maybe, this data will be copy somewhere else and saved as delta in that new location only?

TIA

Re: delta table storage

-werners- — Mon, 30 May 2022 07:55:39 GMT

If you do not define any storage yourself, data is stored as managed tables, meaning in the blob storage of the databricks subscription (which resides on the cloud provider you use).

If you use your own blob storage/data lake, you can (don't have to but you can) write your data there, as unmanaged tables.

But basically you can store it anywhere you want in the cloud, as long as databricks can access it.

DBFS is a semantic layer on top of actual storage, to make working with files more easy.

So if you mounted 3 blob storage f.e., you can write to any of these 3.

Converting to delta:

https://docs.microsoft.com/en-us/azure/databricks/spark/latest/spark-sql/language-manual/delta-convert-to-delta.

But you could also choose to write to another location so data is copied and saved in delta lake format.

Re: delta table storage

Braxx — Mon, 30 May 2022 15:01:03 GMT

thanks, very helpful