Hi @Rishitha , If you've encountered an error indicating that the monotonicallyIncreasingId() function is not supported for streaming DataFrames or datasets, you can use the row_number() window function to generate a monotonically increasing id column for your streaming tables.
Here's how you can do it:
from pyspark.sql import functions as F
from pyspark.sql import Window
# Assuming `streaming_df` is your streaming DataFrame
# Here's how to add a monotonically increasing id column
streaming_df_with_id = streaming_df.withColumn("id", F.row_number().over(Window.orderBy(F.monotonically_increasing_id())))
# Start your streaming query...
console_output_stream = streaming_df_with_id.writeStream.trigger(processingTime="10 seconds").format("console").start()
In this example, the row_number() window function is used to add a monotonically increasing id column to your streaming_df. The orderBy() method utilizes monotonically_increasing_id() as the ordering criteria, creating a unique and increasing id for each row in your streaming DataFrame.
After adding the id column, you can proceed to start your streaming query as usual, and the row_number() function will generate a monotonically increasing id for each row in your streaming DataFrame.
