https://community.databricks.com/t5/get-started-discussions/unable-to-log-metrics-to-table-driver-using-spark/m-p/53942#M1869 <P>Hi Everyone,</P><P>Hope everyone is enjoying their holiday time.</P><P>I have spark streaming job (copy activity from location A to location B) and a Query Listener and attached to the streaming job.</P><P>As mentioned over the attachments I can see&nbsp;numInputRows gets logged to driver logs. But I can't write the&nbsp;finaldf to delta table and even can't log the dataframe to driver logs when I do "show".</P><P>I remember the same piece code used to work on earlier versions may be at spark 3.* which was an year back.</P><P>Please let me know if I am doing something wrong or suggest me if changes required.</P><P>Thanks</P><P>Harish Koduru</P> Mon, 27 Nov 2023 11:40:44 GMT harry546 2023-11-27T11:40:44Z https://community.databricks.com/t5/get-started-discussions/unable-to-log-metrics-to-table-driver-using-spark/m-p/53942#M1869 <P>Hi Everyone,</P><P>Hope everyone is enjoying their holiday time.</P><P>I have spark streaming job (copy activity from location A to location B) and a Query Listener and attached to the streaming job.</P><P>As mentioned over the attachments I can see&nbsp;numInputRows gets logged to driver logs. But I can't write the&nbsp;finaldf to delta table and even can't log the dataframe to driver logs when I do "show".</P><P>I remember the same piece code used to work on earlier versions may be at spark 3.* which was an year back.</P><P>Please let me know if I am doing something wrong or suggest me if changes required.</P><P>Thanks</P><P>Harish Koduru</P> Mon, 27 Nov 2023 11:40:44 GMT https://community.databricks.com/t5/get-started-discussions/unable-to-log-metrics-to-table-driver-using-spark/m-p/53942#M1869 harry546 2023-11-27T11:40:44Z