topic Re: Datadog Installation in Data Engineering

Datadog Installation

Kayla — Mon, 17 Jun 2024 17:36:17 GMT

Is anyone familiar with installing the Datadog agent on clusters? We're not having much luck.
We honestly might not be having the init script run since we're not seeing it in the log, but we can get just a generic "hellow world" init script to run and show in the log with no issue.

Re: Datadog Installation

Walter_C — Wed, 19 Jun 2024 19:08:20 GMT

Is there any specific error you are receiving when running the init script? Does the run complete start up or fail due to the init script?

Re: Datadog Installation

Kayla — Thu, 20 Jun 2024 13:07:54 GMT

I'm not seeing any errors. The cluster does start up without issue.

Re: Datadog Installation

Kayla — Fri, 12 Jul 2024 11:13:31 GMT

Responding here with the solution I found. Hopefully it'll help anyone with similar issues.

First, the Datadog install script is practically a matryoshka doll- the script creates another script which creates a YAML file.
One of the consequences of that is that nothing from the original init script goes into the Databricks logs for init scripts. Instead, it writes its output to /tmp/datadog_start.log. The 2nd script that is created is also in the same directory at /tmp/start_datadog.sh, and I was able to use that to run it directly in a notebook and see the root cause of my error. The error itself was purely a syntax error, easily rectified when I was able to track it down.