Databricks

krishnakash · 11-01-2021

How to properly configure the jar containing the class and spark plugin in Databricks?During DBR 7.3 cluster creation, I tried by setting the spark.plugins, spark.driver.extraClassPath and spark.executor.extraClassPath Spark configs by copying the ja...

krishnakash · 10-04-2021

I have created custom UDF's that generate logs. These logs can be flushed by calling another API exposed which is exposed by an internal layer. However I want to call this API just after the execution of the UDF comes to an end. Is there any way of d...

krishnakash · 10-07-2021

We tried adding the SparkListener, we had added loggers for all type of function of the SparkListenerInterface however according to our observation, we saw that the logs are getting generated in the driver logs. That means the driver node executes th...

Databricks

User Stats

User Activity

How to provide custom class extending SparkPlugin/ExecutorPlugin in Databricks 7.3?

Is there any way of determining last stage of SparkSQL Application Execution?

Re: Is there any way of determining last stage of SparkSQL Application Execution?