cancel
Showing results for 
Search instead for 
Did you mean: 
Data Engineering
cancel
Showing results for 
Search instead for 
Did you mean: 

Forum Posts

JordanYaker
by Contributor
  • 563 Views
  • 0 replies
  • 0 kudos

Integration options for Databricks Jobs and DataDog?

I know that there is already the Databricks (technically Spark) integration for DataDog. Unfortunately, that integration only covers the cluster execution itself and that means only Cluster Metrics and Spark Jobs and Tasks. I'm looking for somethin...

  • 563 Views
  • 0 replies
  • 0 kudos
Hubert-Dudek
by Esteemed Contributor III
  • 671 Views
  • 2 replies
  • 8 kudos

Databricks has added new metrics to its control panel, replacing the outdated Ganglia tool. These new metrics allow users to monitor the following clu...

Databricks has added new metrics to its control panel, replacing the outdated Ganglia tool. These new metrics allow users to monitor the following cluster performance metrics easily:- CPU utilization- Memory usage- Free filesystem space- Network traf...

Screenshot 2023-04-13 154026
  • 671 Views
  • 2 replies
  • 8 kudos
Latest Reply
jose_gonzalez
Moderator
  • 8 kudos

Thank you for sharing @Hubert Dudek​ !!!

  • 8 kudos
1 More Replies
Archana
by New Contributor
  • 2510 Views
  • 3 replies
  • 0 kudos

What are the metrics to be considered for monitoring the Databricks

I am very new to Databricks, just setting up with things. I would like to explore various features of Databricks and start playing around with the environment.I am curious to know what are the metrics should be considered for monitoring the complete ...

  • 2510 Views
  • 3 replies
  • 0 kudos
Latest Reply
jessykoo32
New Contributor II
  • 0 kudos

Databricks is a powerful platform for data engineering, machine learning, and analytics, and it is important to monitor the performance and health of your Databricks environment to ensure that it is running smoothly.Here are a few key metrics that yo...

  • 0 kudos
2 More Replies
Lizz
by New Contributor II
  • 1978 Views
  • 2 replies
  • 3 kudos

Resolved! Forward Spark structured streaming metrics to Datadog

We have a spark streaming application written in Pyspark that we'd like to monitor with Datadog. By default, datadog collects a couple of streaming metrics like 'spark.structured_streaming.processing_rate' and 'spark.structured_streaming.latency'. Ho...

  • 1978 Views
  • 2 replies
  • 3 kudos
Latest Reply
Kaniz
Community Manager
  • 3 kudos

Hi @Liz Zhang​ , We haven't heard from you on the last response from @Shanmugavel Chandrakasu​​, and I was checking back to see if his suggestions helped you. Or else, If you have any solution, please share it with the community as it can be helpful ...

  • 3 kudos
1 More Replies
avinash_goje
by New Contributor II
  • 1689 Views
  • 3 replies
  • 2 kudos

How to send metrics from GCP Databricks to Grafana Cloud through Prometheus?

While connecting the Databricks and Grafana, I have gone through the following approach.Install Grafna Agent in Databrics Clusters from Databricks console --> Not working since the system is not booted with systemd as init systemSince Spark 3 has Pro...

  • 1689 Views
  • 3 replies
  • 2 kudos
Latest Reply
Kaniz
Community Manager
  • 2 kudos

Hi @Avinash Goje​, We haven’t heard from you on the last response from @Hubert Dudek​ , and I was checking back to see if his suggestions helped you. Or else, If you have any solution, please do share that with the community as it can be helpful to o...

  • 2 kudos
2 More Replies
jose_gonzalez
by Moderator
  • 1385 Views
  • 3 replies
  • 0 kudos

How to check my streaming job's metrics?

I would like to know if there is a way to keep track of my running streaming job.

  • 1385 Views
  • 3 replies
  • 0 kudos
Latest Reply
User16869510359
Esteemed Contributor
  • 0 kudos

Streaming metrics are available/exposed mainly through 3 ways:Streaming UI, which is available from Spark 3/DBR 7Streaming listener/Observable metrics APISpark driver logs. Search for the string "Streaming query made progress". The metrics are logged...

  • 0 kudos
2 More Replies
Labels