Unable to log metrics to table/driver using spark StreamingQueryListener
- Mark as New
- Bookmark
- Subscribe
- Mute
- Subscribe to RSS Feed
- Permalink
- Report Inappropriate Content
11-27-2023 03:38 AM - edited 11-27-2023 03:40 AM
Hi Everyone,
Hope everyone is enjoying their holiday time.
I have spark streaming job (copy activity from location A to location B) and a Query Listener and attached to the streaming job.
As mentioned over the attachments I can see numInputRows gets logged to driver logs. But I can't write the finaldf to delta table and even can't log the dataframe to driver logs when I do "show".
I remember the same piece code used to work on earlier versions may be at spark 3.* which was an year back.
Please let me know if I am doing something wrong or suggest me if changes required.
Thanks
Harish Koduru

