Databricks Community

Cluster Policies would help with this not only from a cost management perspective but also standardization of resources across the organization as well simplification for a better user experience. You can find Best Practices on leveraging cluster pol...

You cant save a dataframe without specifying a location. If you are using saveAsTable API then the table will be created in the hive warehouse location. The default location is user.hive.warehouse

A deep clone is recommended way as it holds the history of the table. Also, the DEEP clone is faster than the read-write approach.

If the streaming job is making bling appends to the delta table, then it's perfectly fine to run OPTIMIZE query in parallel.However, if the streaming job is performing MERGE or UPDATE then it can conflict with the OPTIMIZE operations. In such cases w...

DBFS does not have ACL at this point

delta.logRetentionDuration - 30 daysdelta.deletedFileRetentionDuration - 7 days

Using cluster tags we can get the cluster name spark.conf.get("spark.databricks.clusterUsageTags.clusterName")

The files are not physically removed from the Storage by the optimize command. A VACUUM command has to be executed to achieve the same

https://academy.databricks.com/category/self-paced ?

You can run multiple streaming applications in databricks clusters. By default, this would run in the same fair scheduling pool. To enable multiple streaming queries to execute jobs concurrently and to share the cluster efficiently, you can set the q...

We still require a single user to be an owner. But you can set a group to have CAN_MANAGE which unblocks most of the necessary updates. It is released in all Premium workspaces that have Jobs ACLs. The official OWNER is whose identity is used to crea...