Spark eventlog for Cluster pools

[Stephanraj]

Hi,

I would want to setup the cluster logging (to capture eventlogs to /dbfs/cluster-logs dir) in my cluster pool configuration? is it possible?

If I create cluster manually, I could able to setup the cluster logging as mentioned here: https://docs.microsoft.com/en-us/azure/databricks/clusters/configure#cluster-log-delivery

But I am using instance pool to create cluster automatically and run the workload, in that case I was not able to setup the logging and capture the eventlogs. Any help here is highly appreciated. Thanks in advance!!

Thanks,

Stephanraj

[Prabakar]

Hi @Stephanraj C​ instance pool is to reduce cluster start and auto-scaling times for a cluster. Are you using any API to create clusters? If so could you please share the API request?

[Prabakar · Accepted solution]

If you are using API, you can set the logging using the below piece of information in your json body.

https://docs.microsoft.com/en-us/azure/databricks/dev-tools/api/latest/examples#cluster-log-example

"cluster_log_conf": {
        "dbfs": {
            "destination": "dbfs:/cluster-logs"
        }
    },

[Stephanraj]

Hi @Prabakar Ammeappin​ , Thank you so much for your input. Yes it worked well!!

[Prabakar]

Hi @Stephanraj C​ it's good to know that it worked. Please mark the best answer so this question can be closed and will help other members.

[Stephanraj]

Hi @Prabakar Ammeappin​ Thanks. The logs are getting stored in folder with the spark-context-id, is it possible to save the logs with some specific tag names.

[Prabakar]

Hi @Stephanraj C​ I am not sure if that would be possible. Also, I haven't seen such configurations.

[Stephanraj]

Hi @Prabakar Ammeappin​ Okay, I would write some custom script for that. Once again thanks for your support!!