@Someswara Durga Prasad Yaralgadda​ :

I'm glad to hear that you were able to resolve the issue with executing DDL queries in the new runtime version.

Regarding your question about periodically restarting the cluster to improve performance, this is a common practice to prevent long-running clusters from becoming unstable and to ensure that the cluster resources are being utilized effectively. By periodically restarting the cluster, you can release any accumulated resources and refresh the runtime environment, which can help to optimize performance.

However, restarting the cluster manually can be time-consuming and may cause disruptions to ongoing workloads. Databricks provides an automatic cluster termination feature that allows you to specify a time or duration for your cluster to be active, after which the cluster will automatically terminate. This feature can help you to save costs by ensuring that your clusters are only running when they are needed, and it can also help to ensure that the cluster is refreshed periodically to optimize performance.

To enable automatic cluster termination, you can navigate to the Cluster Settings in the Databricks workspace and select the "Auto Termination" option. From there, you can specify the maximum idle time or duration for the cluster, after which the cluster will be terminated automatically.

In addition to automatic cluster termination, you may also consider optimizing your code and workloads to reduce the memory and compute resources required by the cluster. This can include techniques such as data pruning, caching, and partitioning, as well as optimizing your code and queries to reduce the amount of data that needs to be processed. By optimizing your workloads and resources, you can help to ensure that your clusters are running efficiently and cost-effectively.