Power BI Refresh Fails with "Unable to Continue Fetch After Reconnect" from Azure Databricks

[Raj_DB]

Hi everyone,

I'm refreshing a report with dataset (~10 million rows) from Azure Databricks in power Bi service. The refresh runs for about an hour and then fails with: Unable to continue fetch after reconnect. Retry limit exceeded. I am currently unable to determine if this issue originates in Power BI or Databricks. I have tested several potential solutions across both platforms without success. Upon reviewing the SQL warehouse query logs following the report failure, I observed the following error:

SQL warehouse current configuration:

Compute Type: Serverless

cluster size: Medium

I am getting the below error in Power bi service:

Data source error: {
"error": {
"code": "DM_GWPipeline_Gateway_MashupDataAccessError",
"pbi.error": {
"code": "DM_GWPipeline_Gateway_MashupDataAccessError",
"parameters": {},
"details": [
{
"code": "DM_ErrorDetailNameCode_UnderlyingErrorMessage",
"detail": {
"type": 1,
"value": "[Microsoft][Hardy] (35) Error from server: Unable to continue fetch after reconnect. Retry limit exceeded."}},
{"code": "DM_ErrorDetailNameCode_UnderlyingHResult",
"detail": {
"type": 1,
"value": "-2147467259"}}],
"exceptionCulprit": 1}}}

Cluster URI: WABI-INDIA-CENTRAL-A-PRIMARY-redirect.analysis.windows.net
Activity ID: 4f41bd35-ef03-48a5-8b07-57c9b554f648
Request ID: 689cf96f-f8d7-c53e-b6d6-f4eb4e4c5d57

Has anyone experienced this issue or found a solution? Any suggestions would be appreciated.

Thank you!

[balajij8]

Hi Raj,

Query execution on Databricks completed in under a second but the failure is occurring during the client-side result streaming phase between Power BI and Azure Databricks.

When importing Power BI streams result chunks over HTTP connections. If a network hiccup or gateway timeout occurs midway through that data fetch, Power BI attempts to reconnect and resume from the existing position but its interrupted, retried and finally dropped in this case.

You can follow below

1. SQL Warehouse Timeout Settings 

• You can increase Auto Stop / Session timeout to 2 hours.

2. Incremental Refresh in Power BI

• Running a full import of 10 million rows on a scheduled basis generally leads to network dropouts in few cases. You can configure Incremental Refresh in Power BI Service so that only new or updated partitions are transferred during daily refreshes.
• You can use Direct Query with Aggregation Tables in Power BI if feasible.

3. Gateway Timeouts

• 23 minutes of continuous result fetching can cause issues in few cases. You can avoid gateway and connect directly to Power BI service if feasible. Change the gateway settings for better transfer performance if used.

4. Optimize Databricks Layout

• You can pre process the query on the Databricks side to maximize fetch speed if feasible. You can setup liquid clustering on tables for best performance.

5. Classic Warehouse

• Classic SQL warehouses with dedicated compute might provide more stable long running connections in few cases. You can keep size to medium but change type of warehouse to classic pro.