Hi Everyone,
Hope everyone is enjoying their holiday time.
I have spark streaming job (copy activity from location A to location B) and a Query Listener and attached to the streaming job.
As mentioned over the attachments I can see numInputRows gets logged to driver logs. But I can't write the finaldf to delta table and even can't log the dataframe to driver logs when I do "show".
I remember the same piece code used to work on earlier versions may be at spark 3.* which was an year back.
Please let me know if I am doing something wrong or suggest me if changes required.
Thanks
Harish Koduru