Hi @rpaschenko ,
Here are some steps you can take to troubleshoot the issue:
1. **Check the Jobs UI:** Since the Spark UI is unavailable, check the Databricks Jobs UI, which provides a visual overview of completed job runs, filterable by run status and time. The default time filter covers the previous 48 hours. This might give some insight into why the job was cancelled.
Source: [Databricks Release Notes](https://docs.databricks.com/release-notes/product/2023/june.html)
2. **Collect Necessary Information:** Before starting the investigation, collect the following information:
- Notebook URL
- Cluster URL
- Consent to run commands
- Time duration at which the error occurred
- Executor log corresponding to that particular time
3. **Check for Common Errors:** Look for common errors such as Java.lang.outOfMemoryError: Java heap space
in the executor logs.
If such an error is found, it could be that the job was cancelled due to a Java Heap space issue.
In this case, it isn't easy to provide a more specific solution without the logs or any error messages.
If the issue persists, contacting Databricks support with the collected information would be best for further assistance.