topic Re: Expose Databricks Cluster Logs to Grafana in Get Started Discussions

Expose Databricks Cluster Logs to Grafana

joaquin12beard — Mon, 25 Nov 2024 20:55:52 GMT

I am trying to send Databricks cluster logs to Grafana using an init_script where I define the system journals to consume. The issue I am facing is that I cannot get the driver logs, standard error, and output to reach Grafana. Is there something specific I need to enable at the cluster level to expose these logs to the system?

These are the journals I defined.

discovery.relabel "journal_relabel_rules" { targets = [] rule { source_labels = ["__journal__systemd_unit"] target_label = "unit" } rule { source_labels = ["__journal__boot_id"] target_label = "boot_id" } rule { source_labels = ["__journal__transport"] target_label = "transport" } rule { source_labels = ["__journal_priority_keyword"] target_label = "level" } }

Re: Expose Databricks Cluster Logs to Grafana

Walter_C — Mon, 25 Nov 2024 23:38:50 GMT

Are you following a guide to have this set up that you can share with us for information? You can also set up cluster log delivery to an S3 destination and read the logs from there?

Re: Expose Databricks Cluster Logs to Grafana

joaquin12beard — Tue, 26 Nov 2024 12:47:53 GMT

@Walter_C I followed this post but used the Alloy binary to send logs directly to Loki as the source, which Grafana then consumes. I chose not to send logs to S3 to avoid aditional work, preferring to send them directly from the cluster. My question is whether I need to configure anything else on the cluster to expose the driver logs and forward them to Grafana.

Re: Expose Databricks Cluster Logs to Grafana

Walter_C — Tue, 26 Nov 2024 14:20:14 GMT

Does it collect any other set of logs, just missing those ones or not collecting any log?

Re: Expose Databricks Cluster Logs to Grafana

joaquin12beard — Tue, 26 Nov 2024 15:13:54 GMT

@Walter_C This is the log that is arriving.