โ06-14-2021 07:46 PM
โ06-18-2021 10:26 AM
Delta itself is a file format, consisting of Parquet files for the actual data, and a JSON transaction log to maintain the ACID transactions, among other benefits. It will live in whatever store (e.g. S3) you choose in AWS.
Data is only duplicated if you choose to write another table, generally as part of data engineering best practices. For example, as you clean and enrich data, you might write another table so the clean data is readily available and removes the need to reprocess the transformations.
โ06-18-2021 05:55 AM
The data will be saved in S3 AWS only , delta in itself doesn't store anything, it is just a supporting format that keeps additional logs to achieve ACID transactions like traditional sql.
โ06-18-2021 10:26 AM
Delta itself is a file format, consisting of Parquet files for the actual data, and a JSON transaction log to maintain the ACID transactions, among other benefits. It will live in whatever store (e.g. S3) you choose in AWS.
Data is only duplicated if you choose to write another table, generally as part of data engineering best practices. For example, as you clean and enrich data, you might write another table so the clean data is readily available and removes the need to reprocess the transformations.
โ06-22-2021 09:03 PM
And to the earlier comment of Delta being an extension of Parquet. You can start with a dataset in Parquet format in S3 and do an in-place conversion to Delta without having to duplicate the data. See - https://docs.databricks.com/spark/latest/spark-sql/language-manual/delta-convert-to-delta.html for details
Join a Regional User Group to connect with local Databricks users. Events will be happening in your city, and you wonโt want to miss the chance to attend and share knowledge.
If there isnโt a group near you, start one and help create a community that brings people together.
Request a New Group