topic Delta live tables straming in Data Engineering

Delta live tables straming

Rishitha — Wed, 25 Oct 2023 00:40:05 GMT

I'm trying to addmonotonicallyIncreasingId() column to a streaming table and I see the following error

Failed to start stream [table_name] in either append mode or complete mode. Append mode error: Expression(s): monotonically_increasing_id() is not supported with streaming DataFrames/Datasets;

Streaming from S3 buckets to databricks.

Can someone please help.

Re: Delta live tables straming

jose_gonzalez — Wed, 01 Nov 2023 17:18:47 GMT

Hi @Rishitha,

Just a friendly follow-up. Have you had a chance to review my colleague's response to your inquiry? Did it prove helpful, or are you still in need of assistance? Your response would be greatly appreciated.

Re: Delta live tables straming

MuthuLakshmi — Tue, 07 Nov 2023 14:10:28 GMT

The error "Failed to start stream [table_name] in either append mode or complete mode. Append mode error: Expression(s): monotonically_increasing_id() is not supported with streaming DataFrames/Datasets" occurs when trying to start the stream in append mode and you are trying to perform an operation that is not supported in Structured Streaming. You can't do row_number in streaming.

You can resolve this issue by applying SQL window functions.

If you are performing an aggregation, you must apply a watermark to the DataFrame if you want to use append mode. The aggregation must have an event-time column, or a window on the event-time column.

Re: Delta live tables straming

Niro — Wed, 28 Feb 2024 02:07:37 GMT

Is aggregations with row_number() combined with a SQL window function and a watermark still supported in Databricks 14.3?