Databricks SQL script slow execution in workflows using serverless
- Mark as New
- Bookmark
- Subscribe
- Mute
- Subscribe to RSS Feed
- Permalink
- Report Inappropriate Content
โ07-11-2024 05:11 AM - edited โ07-11-2024 05:41 AM
I am running a very simple SQL script within a notebook, using an X-Small SQL Serverless warehouse (that is already running). The execution time is different depending on how it's run:
4s if run interactively (and through SQL editor)
26s if run within a workflow as a notebook task
If you look at the query history screenshot below you see the individual queries are taking the same amount of time to execute, but for some reason there's a few seconds delay between end of one query and start of the next query (line) when running through workflow vs interactively (both using SQL serverless warehouse). I tried both current and preview channel and they both behave the same way
I even tried using an all purpose compute cluster (Single node: Standard_DS3_v2 ยท DBR: 15.3) and the times were consistent when running in a workflow vs interactively.
Script
declare or replace start_time timestamp = current_timestamp();
declare or replace end_time timestamp = current_timestamp();
declare or replace v STRUCT<SOURCE_TBL_MAX_TIMESTAMP STRING, TARGET_TBL_MAX_TIMESTAMP STRING, SOURCE_MAX_TIMESTAMP TIMESTAMP, TARGET_MAX_TIMESTAMP TIMESTAMP, MIN_TIME TIMESTAMP, MAX_TIME TIMESTAMP, MAX_INTERVAL_TIME TIMESTAMP, TABLE_SOURCE STRING, TABLE_TARGET STRING, MERGE_STATEMENT STRING, INTERVAL_COLUMN_NAME STRING, STAGING_TABLE_NAME STRING, STAGING_TABLE_CREATE_STATEMENT STRING, FULL_STAGING_TABLE_NAME STRING>;
select current_timestamp();
select v;
select current_timestamp();
select timestampdiff(MILLISECOND , start_time, current_timestamp()) as total_time_elapsed_ms;
Screenshots
Query History
Further tests
Using a SQL file workflow task, running on X-Small SQL serverless, the same script executes as expected (faster than notebook task):
Not sure why the type of workflow task would impact the execution of the SQL script on a serverless warehouse.
- Labels:
-
Workflows
- Mark as New
- Bookmark
- Subscribe
- Mute
- Subscribe to RSS Feed
- Permalink
- Report Inappropriate Content
โ10-28-2024 11:50 AM
So I was told that the Q4 date was incorrect - in fact there is currently no ETA for when this issue will be fixed. It's considered lower priority by Databricks as not enough customers are impacted or have raised this type of an issue. I would recommend @Ememel that you try and raise this issue through your Databricks support team or Microsoft so it increases the priority?
![](/skins/images/B38AF44D4BD6CE643D2A527BE673CCF6/responsive_peak/images/icon_anonymous_message.png)
![](/skins/images/B38AF44D4BD6CE643D2A527BE673CCF6/responsive_peak/images/icon_anonymous_message.png)
- ยซ Previous
-
- 1
- 2
- Next ยป