โ06-14-2021 07:46 PM
โ06-18-2021 10:26 AM
Delta itself is a file format, consisting of Parquet files for the actual data, and a JSON transaction log to maintain the ACID transactions, among other benefits. It will live in whatever store (e.g. S3) you choose in AWS.
Data is only duplicated if you choose to write another table, generally as part of data engineering best practices. For example, as you clean and enrich data, you might write another table so the clean data is readily available and removes the need to reprocess the transformations.
โ06-18-2021 05:55 AM
The data will be saved in S3 AWS only , delta in itself doesn't store anything, it is just a supporting format that keeps additional logs to achieve ACID transactions like traditional sql.
โ06-18-2021 10:26 AM
Delta itself is a file format, consisting of Parquet files for the actual data, and a JSON transaction log to maintain the ACID transactions, among other benefits. It will live in whatever store (e.g. S3) you choose in AWS.
Data is only duplicated if you choose to write another table, generally as part of data engineering best practices. For example, as you clean and enrich data, you might write another table so the clean data is readily available and removes the need to reprocess the transformations.
โ06-22-2021 09:03 PM
And to the earlier comment of Delta being an extension of Parquet. You can start with a dataset in Parquet format in S3 and do an in-place conversion to Delta without having to duplicate the data. See - https://docs.databricks.com/spark/latest/spark-sql/language-manual/delta-convert-to-delta.html for details
Join our fast-growing data practitioner and expert community of 80K+ members, ready to discover, help and collaborate together while making meaningful connections.
Click here to register and join today!
Engage in exciting technical discussions, join a group with your peers and meet our Featured Members.