How should we think about physical data storage when using Delta Lake? Will data be duplicated or saved within AWS ?

[User16826994223]

The data will be saved in S3 AWS only , delta in itself doesn't store anything, it is just a supporting format that keeps additional logs to achieve ACID transactions like traditional sql.

[KBarlow · Accepted solution]

Delta itself is a file format, consisting of Parquet files for the actual data, and a JSON transaction log to maintain the ACID transactions, among other benefits. It will live in whatever store (e.g. S3) you choose in AWS.

Data is only duplicated if you choose to write another table, generally as part of data engineering best practices. For example, as you clean and enrich data, you might write another table so the clean data is readily available and removes the need to reprocess the transformations.

[aladda]

And to the earlier comment of Delta being an extension of Parquet. You can start with a dataset in Parquet format in S3 and do an in-place conversion to Delta without having to duplicate the data. See - https://docs.databricks.com/spark/latest/spark-sql/language-manual/delta-convert-to-delta.html for details