topic Send System Table Logs to ADLS in Data Governance

Send System Table Logs to ADLS

Amber-26 — Wed, 18 Oct 2023 10:22:30 GMT

Hello,

I am working with Unity Catalogue in Azure Databricks. I have enabled the system schemas for my workspace but unable to figure out a way to send these system table logs to ADLS which I have mounted using Azure Databricks Connector.

Can someone help me with that? This documentation just talks about enabling and disabling.

#UnityCatalogue

Re: Send System Table Logs to ADLS

daniel_sahal — Thu, 19 Oct 2023 04:57:20 GMT

@Amber-26
You can setup Audit Log Delivery to your ADLS. See docu below.

https://learn.microsoft.com/en-us/azure/databricks/administration-guide/account-settings/audit-log-delivery

Re: Send System Table Logs to ADLS

Amber-26 — Thu, 19 Oct 2023 12:36:07 GMT

@daniel_sahal Thanks for the response.

Is there any way to send Billable Usage Logs, Pricing Logs and others to ADLS as well? Will be of great help if you could share the documentation for that as well.

Re: Send System Table Logs to ADLS

karthik_p — Thu, 19 Oct 2023 19:28:40 GMT

@Amber-26 please follow below steps in "export Billing data" and create custom queries this is one way. mostly system billing schema that you enabled have fields (sku etc.) should have match with main billing sheet .

looks there is no direct way to store system schema table logs

https://learn.microsoft.com/en-us/azure/azure-arc/data/view-billing-data-in-azure

Re: Send System Table Logs to ADLS

Amber-26 — Fri, 20 Oct 2023 04:52:41 GMT

Hello @karthik_p , thanks for your response. Although there is no way to send schemas logs to ADLS. One way I found is the perform an SQL query on the schema to get data and send it to ADLS as explained in this documentation

Let me know if this is the right way. Thanks!

Re: Send System Table Logs to ADLS

karthik_p — Fri, 20 Oct 2023 15:49:00 GMT

@Amber-26 you can try this approach, also if you want to have graphical representation of everything you can use lakehouse monitoring feature, where after enabling these tables, you can consume them into dashboards and run them and analyze them