Performance issue with the Databricks Add-on for Splunk 1.4.2

[FalkEngelbrecht]

We are currently using the add-on for Databricks in our on-prem Splunk Enterprise environment. The connection goes directly to the cloud without a proxy. Unfortunately, the add-on is very slow and we lose about 20 seconds with every query. We see that most of the time is lost internally in Splunk when the query has left our search head, then we also have the response from Databricks within a second.
We only use the datbricksquery function.
Can everyone help me?

[siennafaleiro]

It’s likely not Databricks but the Splunk add-on causing the delay. The databricksquery command in version 1.4.2 has known performance issues — most of the lag is inside Splunk while parsing results. Try upgrading to the latest add-on (1.5.x+), monitor search head resources, and if you need faster results, consider running scheduled jobs in Databricks and ingesting them into Splunk instead of live queries.