- Mark as New
- Bookmark
- Subscribe
- Mute
- Subscribe to RSS Feed
- Permalink
- Report Inappropriate Content
08-17-2021 12:57 PM
Specifically for write and read streaming data to HDFS or s3 etc. For IoT specific scenario how it performs on time series transactional data. Can we consider delta table as time series table?
- Labels:
-
Delta-lake
-
Streaming spark
Accepted Solutions
- Mark as New
- Bookmark
- Subscribe
- Mute
- Subscribe to RSS Feed
- Permalink
- Report Inappropriate Content
09-20-2021 09:08 AM
Hi @Arindam Halder , Delta lake is more performant compared to a regular parquet table. pls check below for some stats on the performance
https://docs.azuredatabricks.net/_static/notebooks/delta/optimize-python.html
yes, you can use it for time series forecasting. You could watch the below video to see how forecasting is done at scale using Delta lake
you could also find the delta streaming APIs here
https://docs.databricks.com/delta/delta-streaming.html#table-streaming-reads-and-writes
- Mark as New
- Bookmark
- Subscribe
- Mute
- Subscribe to RSS Feed
- Permalink
- Report Inappropriate Content
09-20-2021 09:08 AM
Hi @Arindam Halder , Delta lake is more performant compared to a regular parquet table. pls check below for some stats on the performance
https://docs.azuredatabricks.net/_static/notebooks/delta/optimize-python.html
yes, you can use it for time series forecasting. You could watch the below video to see how forecasting is done at scale using Delta lake
you could also find the delta streaming APIs here
https://docs.databricks.com/delta/delta-streaming.html#table-streaming-reads-and-writes

