topic Re: Spark eventlog for Cluster pools in Data Engineering

Spark eventlog for Cluster pools

Stephanraj — Mon, 12 Sep 2022 11:56:42 GMT

Hi,

I would want to setup the cluster logging (to capture eventlogs to /dbfs/cluster-logs dir) in my cluster pool configuration? is it possible?

If I create cluster manually, I could able to setup the cluster logging as mentioned here: https://docs.microsoft.com/en-us/azure/databricks/clusters/configure#cluster-log-delivery

But I am using instance pool to create cluster automatically and run the workload, in that case I was not able to setup the logging and capture the eventlogs. Any help here is highly appreciated. Thanks in advance!!

Thanks,

Stephanraj

Re: Spark eventlog for Cluster pools

Prabakar — Mon, 12 Sep 2022 12:20:28 GMT

Hi @Stephanraj C instance pool is to reduce cluster start and auto-scaling times for a cluster. Are you using any API to create clusters? If so could you please share the API request?

Re: Spark eventlog for Cluster pools

Prabakar — Mon, 12 Sep 2022 12:27:18 GMT

If you are using API, you can set the logging using the below piece of information in your json body.

https://docs.microsoft.com/en-us/azure/databricks/dev-tools/api/latest/examples#cluster-log-example

"cluster_log_conf": {
        "dbfs": {
            "destination": "dbfs:/cluster-logs"
        }
    },

Re: Spark eventlog for Cluster pools

Stephanraj — Mon, 12 Sep 2022 12:52:54 GMT

Hi @Prabakar Ammeappin , Thank you so much for your input. Yes it worked well!!

Re: Spark eventlog for Cluster pools

Prabakar — Mon, 12 Sep 2022 13:06:12 GMT

Hi @Stephanraj C it's good to know that it worked. Please mark the best answer so this question can be closed and will help other members.

Re: Spark eventlog for Cluster pools

Stephanraj — Mon, 12 Sep 2022 13:10:34 GMT

Hi @Prabakar Ammeappin Thanks. The logs are getting stored in folder with the spark-context-id, is it possible to save the logs with some specific tag names.

Re: Spark eventlog for Cluster pools

Prabakar — Mon, 12 Sep 2022 13:22:33 GMT

Hi @Stephanraj C I am not sure if that would be possible. Also, I haven't seen such configurations.

Re: Spark eventlog for Cluster pools

Stephanraj — Mon, 12 Sep 2022 13:35:00 GMT

Hi @Prabakar Ammeappin Okay, I would write some custom script for that. Once again thanks for your support!!