Hi @tim-mcwilliams,
It sounds like youโre encountering a situation where the notebook cell appears to hang while running a pymc-marketing
model in Databricks, but the code was eventually completed successfully.
Letโs explore some potential reasons for this behaviour:
-
Resource Constraints:
- Check if your Databricks cluster has sufficient resources (CPU, memory, and disk space) to handle the model-fitting process. If the cluster is under-provisioned, it might cause the progress bar to stall even though the code continues executing.
- Consider increasing the cluster resources or using a larger instance type.
-
Concurrency and Parallelism:
- Databricks Notebooks execute cells in parallel by default. If other cells are running concurrently, they might compete for resources and cause the progress bar to hang.
- Try running the model in an isolated notebook or at a time when other cells are not executing.
-
Interrupt Signal Handling:
- The fact that hitting โInterruptโ doesnโt respond suggests that the notebook might not be handling the interrupt signal properly.
- Check if there are any custom signal handlers or other code that interfere with the default behaviour of interrupting a cell.
- You can also try restarting the kernel or detaching the notebook as youโve been doing.
-
Code Execution and Output:
- Since the code completes successfully and dumps the output into a folder, it seems that the actual computation is working as expected.
- Verify that the output files are correct and contain the expected results.
-
Databricks Environment and Dependencies:
- Ensure that all necessary dependencies (including
pymc-marketing
) are correctly installed in your Databricks environment.
- Check for any conflicting libraries or versions that might cause unexpected behavior.
-
Cluster Logs and Monitoring:
- Continue monitoring the cluster logs to see if any specific errors or warnings occur during the execution of the model.
- Look for any patterns or clues that might help identify the issue.
Hopefully, youโll find a solution soon! ๐๐.