topic Persisting query history data in Data Engineering

Persisting query history data

WWoman — Thu, 23 May 2024 22:25:07 GMT

Hello,

I am looking for a way to persist query history data. I have not have direct access to the system tables. I do have access to a query_history view created by selecting from the system.query.history and system.access.audit system tables. I want to create a simple process that will incrementally load the data that I access through the view into a table..

What is the simplest way to achieve this?

Thanks

Re: Persisting query history data

yuriy — Fri, 24 May 2024 04:50:54 GMT

Hi @WWoman you can set up a simple append only DLT pipeline or just schedule a job that will run merge statement insert * only. In both cases you will get a table that will accumulate history of additions in your view over time and in both cases it will be incremental

Re: Persisting query history data

syed_sr7 — Thu, 27 Jun 2024 16:35:12 GMT

Is any system table there for query history?