Databricks Community

If the cluster is unsued for 30 days, Databricks removes the cluster. This is a general clean-up policy. It's possible to whitelist a cluster from this clean-up by Pinning the cluster. https://docs.databricks.com/clusters/clusters-manage.html#pin-a-c...

I think you are writing the data in override mode. what happens in delta is it doesn't delete the data for certain days even it is written by overwrite mode for versioning , and you will be able to query only most recent data,But in format parque...

Delta table is the non-hive compatible format. So there must also be permissions for a client to access the path to the database’s location so that it can create a new temporary “directory” there. This comes from Spark SQL’s handling of external tabl...

Use the below code snippet from a notebook%scala import java.sql.Connection import java.sql.DriverManager import java.sql.ResultSet import java.sql.SQLException   /** * For details on what this query means, checkout https://dev.mysql.com/doc/refma...

Yes, check out the immuta web page on the Databricks Integration. https://www.immuta.com/integrations/databricks

Right now, we do not have an option to measure the compute cost at a query level.

If you want to keep your checkpoints X days, you can set delta.checkpointRetentionDuration to X days this way:spark.sql(f""" ALTER TABLE delta.`path` SET TBLPROPERTIES ( delta.checkpointRetentionDuration = 'X days'...

There is no direct way to track the progress of the VACUUM command. One easy workaround is to run a DRY RUN from another notebook which will give the estimate of files to be deleted at that point in time. This will give a rough estimate of files to b...

Autoloader has an option - "cloudFiles. allowOverwrites". This determines whether to allow input directory file changes to overwrite existing data. This option is available in Databricks Runtime 7.6 and above.