Delta live tables straming
- Mark as New
- Bookmark
- Subscribe
- Mute
- Subscribe to RSS Feed
- Permalink
- Report Inappropriate Content
10-24-2023 05:40 PM
I'm trying to addmonotonicallyIncreasingId() column to a streaming table and I see the following error
Failed to start stream [table_name] in either append mode or complete mode.
Append mode error: Expression(s): monotonically_increasing_id() is not supported with streaming DataFrames/Datasets;
Streaming from S3 buckets to databricks.
Can someone please help.
- Mark as New
- Bookmark
- Subscribe
- Mute
- Subscribe to RSS Feed
- Permalink
- Report Inappropriate Content
11-01-2023 10:18 AM
Hi @Rishitha,
Just a friendly follow-up. Have you had a chance to review my colleague's response to your inquiry? Did it prove helpful, or are you still in need of assistance? Your response would be greatly appreciated.
- Mark as New
- Bookmark
- Subscribe
- Mute
- Subscribe to RSS Feed
- Permalink
- Report Inappropriate Content
11-07-2023 06:10 AM
The error "Failed to start stream [table_name] in either append mode or complete mode. Append mode error: Expression(s): monotonically_increasing_id() is not supported with streaming DataFrames/Datasets" occurs when trying to start the stream in append mode and you are trying to perform an operation that is not supported in Structured Streaming. You can't do row_number in streaming.
You can resolve this issue by applying SQL window functions.
If you are performing an aggregation, you must apply a watermark to the DataFrame if you want to use append mode. The aggregation must have an event-time column, or a window on the event-time column.
- Mark as New
- Bookmark
- Subscribe
- Mute
- Subscribe to RSS Feed
- Permalink
- Report Inappropriate Content
02-27-2024 06:06 PM - edited 02-27-2024 06:07 PM
Is aggregations with row_number() combined with a SQL window function and a watermark still supported in Databricks 14.3?