Databricks Community

 G1GC can solve problems in some cases where garbage collection is a bottleneck. checkout https://databricks.com/blog/2015/05/28/tuning-java-garbage-collection-for-spark-applications.html

You could potentially do this through a Global Init Script - https://docs.databricks.com/clusters/init-scripts.html

Dy doing a `GET` call using the cluster idcurl --netrc -X GET \ https://dbc-a1b2345c-d6e7.cloud.databricks.com/api/2.0/clusters/get \ --data '{ "cluster_id": "1234-567890-myclustID" }' \ | jq .The response json will have a `state` tag which will look...

You could look at S3 Intelligent-Tiering - https://aws.amazon.com/about-aws/whats-new/2018/11/s3-intelligent-tiering/

Can you provide some additional details on this? What components are we comparing the states for?

Deleting the Delta log directory would cause you to lose the underlying transaction history on the delta table and other delta related optimizations. In effect the table would be converted to a Parquet table at that point

Also, a lot of examples here: https://docs.databricks.com/administration-guide/clusters/policies.html#cluster-policy-examples

Global: run on every cluster in the workspace. They can help you to enforce consistent cluster configurations across your workspace. Use them carefully because they can cause unanticipated impacts, like library conflicts. Only admin users can create ...

Yes.Currently supported paths are:BYO VPC -> New BYO VPCDatabricks-created VPC -> New Databricks-created VPC

See this knowledge base article for details - https://kb.databricks.com/clusters/overwrite-log4j-logs.html

For Databricks Runtime 5.5 LTS, Spark jobs, Python notebook cells, and library installation all support both Python 2 and 3.The default Python version for clusters created using the UI is Python 3. In Databricks Runtime 5.5 LTS the default version fo...