cancel
Showing results forย 
Search instead forย 
Did you mean:ย 
Data Engineering
Join discussions on data engineering best practices, architectures, and optimization strategies within the Databricks Community. Exchange insights and solutions with fellow data engineers.
cancel
Showing results forย 
Search instead forย 
Did you mean:ย 

Delta live tables straming

Rishitha
New Contributor III

I'm trying to addmonotonicallyIncreasingId() column to a streaming table and I see the following error

Failed to start stream [table_name] in either append mode or complete mode.
Append mode error: Expression(s): monotonically_increasing_id() is not supported with streaming DataFrames/Datasets;

Streaming from S3 buckets to databricks. 

Can someone please help.

3 REPLIES 3

jose_gonzalez
Databricks Employee
Databricks Employee

Hi @Rishitha,

Just a friendly follow-up. Have you had a chance to review my colleague's response to your inquiry? Did it prove helpful, or are you still in need of assistance? Your response would be greatly appreciated.

MuthuLakshmi
Databricks Employee
Databricks Employee

The error "Failed to start stream [table_name] in either append mode or complete mode. Append mode error: Expression(s): monotonically_increasing_id() is not supported with streaming DataFrames/Datasets" occurs when trying to start the stream in append mode and you are trying to perform an operation that is not supported in Structured Streaming. You can't do row_number in streaming.

You can resolve this issue by applying SQL window functions.

If you are performing an aggregation, you must apply a watermark to the DataFrame if you want to use append mode. The aggregation must have an event-time column, or a window on the event-time column.

Niro
New Contributor II

Is aggregations with row_number() combined with a SQL window function and a watermark still supported in Databricks 14.3?

Connect with Databricks Users in Your Area

Join a Regional User Group to connect with local Databricks users. Events will be happening in your city, and you wonโ€™t want to miss the chance to attend and share knowledge.

If there isnโ€™t a group near you, start one and help create a community that brings people together.

Request a New Group