Hi @NandiniN,

Thank you for your response and insights. I appreciate you taking the time to help me troubleshoot this issue.

To provide more context:

1. DataFrame Details:

   • df_10hz contains high-frequency sensor data, and I am attempting to update its name column based on activity periods from df_enrich_data.
   • df_enrich_data includes enrichment data such as timestamps and activity IDs.

2. Environment:

   • I'm using Databricks serverless compute.
   • The dataset size is relatively large, which may contribute to resource constraints.

3. Error Context:

   • The error specifically occurs when I try to display the df_10hz DataFrame using the display() function.
   • I initially used a loop to iterate through each row of df_enrich_data to update the df_10hz DataFrame conditionally. However, this approach led to the [RETRIES_EXCEEDED] error.
   • To troubleshoot, I tested the same logic with a smaller dataset, and it worked perfectly fine. This suggests that the issue might be related to data volume or resource limitations in the serverless compute environment.

To work around this issue, I replaced the loop with a join operation to update the df_10hz DataFrame. This approach has significantly improved performance and avoided the retry error. While the join resolves the issue, I am curious to understand why the display() function fails with the larger dataset, even after retries, and if there are specific configurations or optimizations for serverless compute that could help.

Based on your suggestion, I will:

• Review the logs to identify what is being retried and determine if there are potential network or resource bottlenecks.
• Continue monitoring resource usage in the serverless environment to ensure it meets the workload demands.

Do you have any additional recommendations for optimizing large DataFrame operations in serverless compute or handling display() errors with large datasets?

Thank you again for your guidance!