cancel
Showing results for 
Search instead for 
Did you mean: 
Administration & Architecture
Explore discussions on Databricks administration, deployment strategies, and architectural best practices. Connect with administrators and architects to optimize your Databricks environment for performance, scalability, and security.
cancel
Showing results for 
Search instead for 
Did you mean: 

SQL warehouse realtime monitoring

Jinyoung
New Contributor

A few days ago, our SQL warehouse had been back pressure and I knew that issue a hour later.
I really wanna know how to get status of databricks SQL warehouse such as using DataDog agent not using web UI.

2 REPLIES 2

Kaniz_Fatma
Community Manager
Community Manager

Hi @Jinyoung , Certainly! Monitoring your Databricks SQL warehouse using Datadog is a great approach. Let’s explore how you can achieve this:

 

Deploy Datadog to Your Databricks cluster:

  • Datadog provides an integration for Databricks that unifies infrastructure metrics, logs, and Spark performance metrics.
  • You can run the following code in a Databricks notebook to generate an installation script for attaching Datadog to your cluster:
  • Replace 'YOUR_API_KEY', 'YOUR_APP_KEY', and 'YOUR_DASHBOARD_ID' with your actual Datadog credentials and dashboard ID.
  • The script will install the Datadog Agent on your Databricks cluster when it starts up.

Collect Metrics and Logs:

  • Datadog will collect resource metrics (e.g., memory usage, CPU load) from the nodes in your clusters.
  • These metrics are automatically tagged with the cluster name, allowing you to examine resource usage across specific clusters.
  • You can track the health of your Databricks clusters, fine-tune Spark jobs, and troubleshoot issues.

View Metrics and Optimize:

  • Use Datadog’s out-of-the-box dashboard to view detailed system metrics from your cluster infrastructure.
  • Additionally, monitor Spark metrics via Datadog’s Spark integration.
  • Make informed decisions based on real-time visibility into the health of your nodes and job performance.
  • Optimize your clusters by adjusting configuration and application code.

Remember that monitoring infrastructure resource metrics is crucial for ensuring your clusters are correctly sized for the jobs you’re running. Datadog helps you identify bottlenecks, optimize performance, and troubleshoot effectively.

Feel free to explore Datadog’s documentation for more details on setting up and configuring the Databricks integration. If you have any further questions, feel free to ask! 🚀

Yev
New Contributor II

Hi There! Is there any way to integrate Datadog with Databricks SQL Warehouse? I only found documentation related to clusters that seem to be different than SQL Warehouse in Databricks. However, I tried to add a global init script for Datadog integration (as Datadog recommended) and rebooted my SQL Warehouse Classic. After waiting for a while, I didn't see any metrics in my Datadog dashboard but I can see metrics for my newly created cluster machine in Databricks.
I would appreciate any suggestions about it.

Best regards,
Yevhenii

Join 100K+ Data Experts: Register Now & Grow with Us!

Excited to expand your horizons with us? Click here to Register and begin your journey to success!

Already a member? Login and join your local regional user group! If there isn’t one near you, fill out this form and we’ll create one for you to join!