https://community.databricks.com/t5/data-engineering/integration-options-for-databricks-jobs-and-datadog/m-p/3863#M761 <P>I know that there is already the Databricks (technically Spark) integration for DataDog. Unfortunately, that integration only covers the cluster execution itself and that means only Cluster Metrics and <I>Spark</I> Jobs and Tasks. I'm looking for something that will allow me to track metrics about the Databricks Jobs (e.g., Successful Job Runs, Failed Tasks, etc.). </P><P></P><P>Currently, it seems like my only option would be to use a combination of webhook notifications for the job and custom code inside the task. Unfortunately, this has the following drawbacks:</P><P></P><OL><LI>We use PySpark and if I get a Kernel Unresponsive error, then I'm not going to be able to report metrics with custom code.</LI><LI>The web hook notifications don't produce events that are ready for use as custom metrics within DataDog so I will have to create some sort of custom handler for mapping/enriching the events from the web hook.</LI></OL><P></P><P>I feel like this isn't a unique challenge and that I'm not the only one who would look for something like this, so before I go down the rabbit hole of crafting something home-brew to solve this problem, I thought I would check with the community/Databricks support to see if I'm missing something.</P> Tue, 30 May 2023 18:47:28 GMT JordanYaker 2023-05-30T18:47:28Z https://community.databricks.com/t5/data-engineering/integration-options-for-databricks-jobs-and-datadog/m-p/3863#M761 <P>I know that there is already the Databricks (technically Spark) integration for DataDog. Unfortunately, that integration only covers the cluster execution itself and that means only Cluster Metrics and <I>Spark</I> Jobs and Tasks. I'm looking for something that will allow me to track metrics about the Databricks Jobs (e.g., Successful Job Runs, Failed Tasks, etc.). </P><P></P><P>Currently, it seems like my only option would be to use a combination of webhook notifications for the job and custom code inside the task. Unfortunately, this has the following drawbacks:</P><P></P><OL><LI>We use PySpark and if I get a Kernel Unresponsive error, then I'm not going to be able to report metrics with custom code.</LI><LI>The web hook notifications don't produce events that are ready for use as custom metrics within DataDog so I will have to create some sort of custom handler for mapping/enriching the events from the web hook.</LI></OL><P></P><P>I feel like this isn't a unique challenge and that I'm not the only one who would look for something like this, so before I go down the rabbit hole of crafting something home-brew to solve this problem, I thought I would check with the community/Databricks support to see if I'm missing something.</P> Tue, 30 May 2023 18:47:28 GMT https://community.databricks.com/t5/data-engineering/integration-options-for-databricks-jobs-and-datadog/m-p/3863#M761 JordanYaker 2023-05-30T18:47:28Z https://community.databricks.com/t5/data-engineering/integration-options-for-databricks-jobs-and-datadog/m-p/113159#M44444 <P>Did you ever figure out a good solution? I'm faced with the same problem.</P> Thu, 20 Mar 2025 14:23:18 GMT https://community.databricks.com/t5/data-engineering/integration-options-for-databricks-jobs-and-datadog/m-p/113159#M44444 BrenAlex_88575 2025-03-20T14:23:18Z https://community.databricks.com/t5/data-engineering/integration-options-for-databricks-jobs-and-datadog/m-p/135529#M50375 <P>Personally, I'm using their Data Jobs Monitoring product&nbsp;<A href="https://docs.datadoghq.com/data_jobs/databricks/" target="_self">https://docs.datadoghq.com/data_jobs/databricks/</A>&nbsp;that works great and gives the right insights both for my high level job execution stats and Spark deeper metrics</P> Tue, 21 Oct 2025 15:16:48 GMT https://community.databricks.com/t5/data-engineering/integration-options-for-databricks-jobs-and-datadog/m-p/135529#M50375 greg-0935 2025-10-21T15:16:48Z