cancel
Showing results for 
Search instead for 
Did you mean: 
Data Engineering
Join discussions on data engineering best practices, architectures, and optimization strategies within the Databricks Community. Exchange insights and solutions with fellow data engineers.
cancel
Showing results for 
Search instead for 
Did you mean: 

Datadog Installation

Kayla
Contributor III

Is anyone familiar with installing the Datadog agent on clusters? We're not having much luck. 
 We honestly might not be having the init script run since we're not seeing it in the log, but we can get just a generic "hellow world" init script to run and show in the log with no issue.

1 ACCEPTED SOLUTION

Accepted Solutions

Kayla
Contributor III

Responding here with the solution I found. Hopefully it'll help anyone with similar issues.

First, the Datadog install script is practically a matryoshka doll- the script creates another script which creates a YAML file.
One of the consequences of that is that nothing from the original init script goes into the Databricks logs for init scripts. Instead, it writes its output to /tmp/datadog_start.log. The 2nd script that is created is also in the same directory at /tmp/start_datadog.sh, and I was able to use that to run it directly in a notebook and see the root cause of my error. The error itself was purely a syntax error, easily rectified when I was able to track it down.

View solution in original post

3 REPLIES 3

Walter_C
Honored Contributor
Honored Contributor

Is there any specific error you are receiving when running the init script? Does the run complete start up or fail due to the init script?

Kayla
Contributor III

I'm not seeing any errors. The cluster does start up without issue.

Kayla
Contributor III

Responding here with the solution I found. Hopefully it'll help anyone with similar issues.

First, the Datadog install script is practically a matryoshka doll- the script creates another script which creates a YAML file.
One of the consequences of that is that nothing from the original init script goes into the Databricks logs for init scripts. Instead, it writes its output to /tmp/datadog_start.log. The 2nd script that is created is also in the same directory at /tmp/start_datadog.sh, and I was able to use that to run it directly in a notebook and see the root cause of my error. The error itself was purely a syntax error, easily rectified when I was able to track it down.

Join 100K+ Data Experts: Register Now & Grow with Us!

Excited to expand your horizons with us? Click here to Register and begin your journey to success!

Already a member? Login and join your local regional user group! If there isn’t one near you, fill out this form and we’ll create one for you to join!