Hi Databricks Community,

I am encountering an issue when trying to display a DataFrame in a Python notebook using serverless compute. The operation seems to fail after several retries, and I get the following error message:

[RETRIES_EXCEEDED] The maximum number of retries has been exceeded.  
File /databricks/python/lib/python3.10/site-packages/pyspark/sql/connect/client/core.py:1435, in SparkConnectClient._analyze(self, method, **kwargs)  
   1434 with attempt:  
-> 1435     resp = self._stub.AnalyzePlan(req, metadata=self.metadata())  
   1436     self._verify_response_integrity(resp)  
File /databricks/python/lib/python3.10/site-packages/pyspark/sql/connect/client/retries.py:236, in Retrying._wait(self)  
    234 # Exceeded retries  
    235 logger.debug(f"Given up on retrying. error: {repr(exception)}")  
--> 236 raise RetriesExceeded(error_class="RETRIES_EXCEEDED", message_parameters={}) from exception  

Here are some additional details:

• Environment: Python notebook using Databricks serverless compute.
• Code example:

from functools import reduce
from pyspark.sql import DataFrame
from pyspark.sql.functions import col, when, lit
from pyspark.sql.types import StringType, TimestampType
from tqdm import tqdm

df_10hz = df_10hz.withColumn('name', lit(None).cast(StringType()))

# Loop through each row in activity_periods and filter sensor_data
for row in tqdm(df_enrich_data.collect(), desc="Processing activity"):
    period_end = row['Timestamp']
    act_id = row['actId']

    # Debug print messages
    print(f"Processing actId: {act_id}")
    
    # Update Name column based on conditions
    df_10hz = df_10hz.withColumn("name", when(
        (col("actId") == activity_id), lit(row['name'])).otherwise(col("name")))

display(df_10hz)

 Has anyone else encountered this issue? We would greatly appreciate any tips on how to resolve or debug it further!

Thank you in advance for your help!