Usage of SparkMetric_CL, SparkListenerEvent_CL and SparkLoggingEvent_CL

[groch_adam]

I am wondering If can retrieve any information from Azure Log Analytics custom tables (already set) for Azure Databricks. Would like to retrieve information about query and data performance for SQL Warehouse Cluster. I am not sure If I can get it from custom log analytics table (or it is only for Compute Cluster).

[Kaniz_Fatma]

Hi @groch_adam, Retrieving information from Azure Log Analytics custom tables for Azure Databricks is possible.

Let me guide you through the process.

1. Azure Databricks Monitoring Library:

   • To send application logs and metrics from Azure Databricks to a Log Analytics workspace, you can use the Azure Databricks Monitoring Library. This library streams Apache Spark™ level events and Spark Structured Streaming metrics from your jobs to Azure Monitor.
   • First, configure your Azure Databricks cluster to use the monitoring library. You’ll find detailed instructions in the GitHub readme.
   • Note that the monitoring library supports Azure Databricks Runtimes 10.x (Spark 3.2.x) and earlier, ...¹.

2. Custom Log Tables in Azure Monitor:

   • You can create custom tables in Azure Monitor Logs to store specific data. These tables allow you to organize and query your data effectively.
   • To create a custom table in the Azure portal:
     • Go to your Log Analytics workspace.
     • Select “Tables” from the menu.
     • Click “Create” and then choose “New custom log (DCR-based).”
     • Specify a name for the table (you don’t need to add the _CL suffix; it’s added automatically)².

3. Querying Data from Azure Log Analytics:

   • Once you’ve set up the monitoring library and created custom tables, you can query your data:
     • Azure Portal: You can view and query your data directly from the Azure portal. Go to your Log Analytics Workspace, select “Logs,” and start writing queries to retrieve the data you need.
     • REST API: You can programmatically retrieve data from your workspace using the Log Analytics REST AP...³.

4. Specifically for SQL Warehouse Cluster:

   • If you’re interested in query and data performance for your SQL Warehouse Cluster, you might want to focus on specific metrics related to your workload. These could include execution times, resource utilization, and query execution plans.
   • Consider creating custom log tables that capture relevant information specific to your SQL Warehouse Cluster. You can then query these tables to gain insights into performance.

Remember to tailor your custom tables and queries to match the specific information you need for your use case. If you encounter any issues or need further assistance, feel free to ask! 😊🚀¹²³

^{To ensure we provide you with the best support, could you please take a moment to review the response and choose the one that best answers your question? Your feedback not only helps us assist you better but also benefits other community members who may have similar questions in the future.}

^{If you found the answer helpful, consider giving it a kudo. If the response fully addresses your question, please mark it as the accepted solution. This will help us close the thread and ensure your question is resolved.}

^{We appreciate your participation and are here to assist you further if you need it!}