Databricks Add-on for Splunk v1.2 - Error in 'databricksquery' command

[hukel]

Is anyone else using the new v1.2 of the Databricks Add-on for Splunk ?   We upgraded to 1.2 and now get this error for all queries.

Running process: /opt/splunk/bin/nsjail-wrapper /opt/splunk/bin/python3.7 /opt/splunk/etc/apps/TA-Databricks/bin/databricksquery.py
Error in 'databricksquery' command: External search command exited unexpectedly with non-zero error code 1.

I've opened an issue here https://github.com/databrickslabs/splunk-integration/issues/42 but haven't gotten a follow-up.  

Is anyone else using this add-on successfully with v1.2?

[shan_chandra]

@hukel - could you please share the full error stack trace?

[hukel]

I can't see the python stack trace because the TA doesn't output it to a Splunk-logged location (that I can find).    The search.log output is all I can see (pasted below).

08-07-2023 16:03:05.046 INFO  SearchParser [994756 searchOrchestrator] - PARSING: | databricksquery command_timeout=1200 query="\n     \n     SELECT ImageFileName, _time, *\n     FROM silver.ProcessRollup2 \n\n     WHERE event_date BETWEEN '2023-08-07' AND '2023-08-07'\n       AND _time >= 1691409780.000 AND _time <= 1691424183.000\n       AND (\n          LOWER(ImageFileName) LIKE '\\\\\\\\device\\\\\\\\harddiskvolume%\\\\\\\\\agentexecutor.exe'\n       )\n     ORDER BY _time DESC \n\n     LIMIT 1 "
08-07-2023 16:03:05.047 INFO  ServerConfig [994756 searchOrchestrator] - Will add app jailing prefix /opt/splunk/bin/nsjail-wrapper for TA-Databricks
08-07-2023 16:03:05.047 INFO  ChunkedExternProcessor [994756 searchOrchestrator] - Running process: /opt/splunk/bin/nsjail-wrapper /opt/splunk/bin/python3.7 /opt/splunk/etc/apps/TA-Databricks/bin/databricksquery.py
08-07-2023 16:03:05.747 INFO  ChunkedExternProcessor [994756 searchOrchestrator] - Custom search command is a generating command.
08-07-2023 16:03:05.747 WARN  ChunkedExternProcessor [994756 searchOrchestrator] - Error adding inspector message: invalid level or message already exists
08-07-2023 16:03:05.747 INFO  SearchPipeline [994756 searchOrchestrator] - ReportSearch=0 AllowBatchMode=0
08-07-2023 16:03:05.747 INFO  SearchPhaseGenerator [994756 searchOrchestrator] - No need for RTWindowProcessor
08-07-2023 16:03:05.747 INFO  SearchPhaseGenerator [994756 searchOrchestrator] - Adding timeliner to final phase
08-07-2023 16:03:05.747 INFO  SearchParser [994756 searchOrchestrator] - PARSING: | timeliner remote=0 partial_commits=0 max_events_per_bucket=10000 fieldstats_update_maxperiod=60 bucket=0 extra_field=*
08-07-2023 16:03:05.747 INFO  TimelineCreator [994756 searchOrchestrator] - Creating timeline with remote=0 partialCommits=0 commitFreq=0 syncKSFreq=0 maxSyncKSPeriodTime=60000 bucket=0 latestTime=1691424183.000000 earliestTime=1691409780.000000
08-07-2023 16:03:05.747 INFO  SearchPhaseGenerator [994756 searchOrchestrator] - required fields list to add to different pipelines = *,_bkt,_cd,_si,host,index,linecount,source,sourcetype,splunk_server
08-07-2023 16:03:05.747 INFO  SearchPhaseGenerator [994756 searchOrchestrator] - Search Phases created.
08-07-2023 16:03:05.749 INFO  SearchOrchestrator [994756 searchOrchestrator] - Starting the status control thread.
08-07-2023 16:03:05.749 INFO  SearchOrchestrator [994756 searchOrchestrator] - Starting phase=1
08-07-2023 16:03:05.749 INFO  ReducePhaseExecutor [994794 phase_1] - Starting phase_1
08-07-2023 16:03:05.749 INFO  SearchStatusEnforcer [994787 StatusEnforcerThread] - Enforcing disk quota = 10485760000
08-07-2023 16:03:05.805 ERROR ChunkedExternProcessor [994794 phase_1] - EOF while attempting to read transport header read_size=0
08-07-2023 16:03:05.805 ERROR ChunkedExternProcessor [994794 phase_1] - Error in 'databricksquery' command: External search command exited unexpectedly with non-zero error code 1.
08-07-2023 16:03:05.805 INFO  ReducePhaseExecutor [994794 phase_1] - Ending phase_1

[shan_chandra]

@hukel -  Does the below query runs fine in an isolated notebook?

SELECT ImageFileName, _time, *\n     FROM silver.ProcessRollup2 \n\n     WHERE event_date BETWEEN '2023-08-07' AND '2023-08-07'\n       AND _time >= 1691409780.000 AND _time <= 1691424183.000\n       AND (\n          LOWER(ImageFileName) LIKE '\\\\\\\\device\\\\\\\\harddiskvolume%\\\\\\\\\agentexecutor.exe'\n       )\n     ORDER BY _time DESC \n\n     LIMIT 1 

[hukel]

Yes,  this is a test query that I always use.  It has only stopped working after the 1.2 upgrade.

[hukel · Accepted solution]

There is a new mandatory parameter for databricksquery called account_name.    This breaking change is not documented in Splunkbase release notes but it does appear in the docs within the Splunk app.

databricksquery cluster="<cluster_name>" query="<SQL_query>" command_timeout=<timeout_in_seconds> account_name="<account_name>"