Databricks Community

If the streaming job is making bling appends to the delta table, then it's perfectly fine to run OPTIMIZE query in parallel.However, if the streaming job is performing MERGE or UPDATE then it can conflict with the OPTIMIZE operations. In such cases w...

delta.logRetentionDuration - 30 daysdelta.deletedFileRetentionDuration - 7 days

Using cluster tags we can get the cluster name spark.conf.get("spark.databricks.clusterUsageTags.clusterName")

The files are not physically removed from the Storage by the optimize command. A VACUUM command has to be executed to achieve the same

https://academy.databricks.com/category/self-paced ?

You can run multiple streaming applications in databricks clusters. By default, this would run in the same fair scheduling pool. To enable multiple streaming queries to execute jobs concurrently and to share the cluster efficiently, you can set the q...

We still require a single user to be an owner. But you can set a group to have CAN_MANAGE which unblocks most of the necessary updates. It is released in all Premium workspaces that have Jobs ACLs. The official OWNER is whose identity is used to crea...

Only HTTPS is supported right now.If SSH is required for your use case, please let your Databricks Rep know and reference the Idea DB-I-3697 so that it can be prioritized.

You can clone any repo, the security concern is usually around proprietary code exfiltration, whether intentional or accidental.

It's not currently on the roadmap but please create an Idea and Product team could consider it, based on demand.https://ideas.databricks.com/

Feature table deletion is a potentially dangerous operation, since downstream consumers of feature tables (models, online stores, jobs, etc) may break due to the deletion. We might support a safe way to do this in future. In the meanwhile, we may be ...