Can you share what command were you executing ? Also were you currently doing exception handling within your code with try/catch exception ? Can you also check the driver logs during the time error happened ? The driver logs should have more detail...
Have you checked this https://docs.databricks.com/api-explorer/workspace/queryhistory/list you can list the queries based on time range as well. So you can try passing the fields in the filter_by parameter. Then pass the value as (current time - 5 m...
This is most likely because more files became eligible for compaction (optimize). By default there is a limit of 50 files or so per partition, below which the partition doesn't qualify for optimize. Only if there are 50+ files within a partition the...
Column-specific access without dynamic views is currently in private preview. You can work with Databricks accounts team to sign up for a private preview to get an early access. Once this is in GA, it will be generally available. Hope it clarifies.
