topic Re: How should we think about physical data storage when using Delta Lake? Will data be duplicated or saved within AWS ? in Data Engineering

How should we think about physical data storage when using Delta Lake? Will data be duplicated or saved within AWS ?

Anonymous — Tue, 15 Jun 2021 02:46:59 GMT

Re: How should we think about physical data storage when using Delta Lake? Will data be duplicated or saved within AWS ?

User16826994223 — Fri, 18 Jun 2021 12:55:18 GMT

The data will be saved in S3 AWS only , delta in itself doesn't store anything, it is just a supporting format that keeps additional logs to achieve ACID transactions like traditional sql.

Re: How should we think about physical data storage when using Delta Lake? Will data be duplicated or saved within AWS ?

KBarlow — Fri, 18 Jun 2021 17:26:47 GMT

Delta itself is a file format, consisting of Parquet files for the actual data, and a JSON transaction log to maintain the ACID transactions, among other benefits. It will live in whatever store (e.g. S3) you choose in AWS.

Data is only duplicated if you choose to write another table, generally as part of data engineering best practices. For example, as you clean and enrich data, you might write another table so the clean data is readily available and removes the need to reprocess the transformations.

Re: How should we think about physical data storage when using Delta Lake? Will data be duplicated or saved within AWS ?

aladda — Wed, 23 Jun 2021 04:03:41 GMT

And to the earlier comment of Delta being an extension of Parquet. You can start with a dataset in Parquet format in S3 and do an in-place conversion to Delta without having to duplicate the data. See - https://docs.databricks.com/spark/latest/spark-sql/language-manual/delta-convert-to-delta.html for details