https://community.databricks.com/t5/data-engineering/any-recommendations-to-reduce-the-cluster-spin-up-time/m-p/23051#M15869 <P>Databricks pool concept may help reduce cluster start up time. more details are <A href="https://docs.databricks.com/clusters/instance-pools/index.html" alt="https://docs.databricks.com/clusters/instance-pools/index.html" target="_blank">here</A></P> Thu, 17 Jun 2021 20:15:48 GMT Srikanth_Gupta_ 2021-06-17T20:15:48Z https://community.databricks.com/t5/data-engineering/any-recommendations-to-reduce-the-cluster-spin-up-time/m-p/23049#M15867 Thu, 17 Jun 2021 18:59:07 GMT https://community.databricks.com/t5/data-engineering/any-recommendations-to-reduce-the-cluster-spin-up-time/m-p/23049#M15867 Anonymous 2021-06-17T18:59:07Z https://community.databricks.com/t5/data-engineering/any-recommendations-to-reduce-the-cluster-spin-up-time/m-p/23050#M15868 <P>There are a few approaches to consider here. </P><P></P><P>There is an interesting session in the 2021 Data &amp; AI summit on Nephos -which implements Lakehouse without the infrastructure management overheads. You could <A href="https://databricks.com/session_na21/run-your-queries-instantly-in-one-of-the-most-optimized-environments" alt="https://databricks.com/session_na21/run-your-queries-instantly-in-one-of-the-most-optimized-environments" target="_blank">watch it here </A>. Clusters in this platform would spin up significantly faster since databricks would be managing underlying VMS</P><P></P><P>For compute in customer's environment, there is a feature called <A href="https://docs.databricks.com/clusters/instance-pools/index.html" alt="https://docs.databricks.com/clusters/instance-pools/index.html" target="_blank">Cluster pools</A> which maintains a set of idle, ready-to-use VM instances</P><P></P><P>Optionally, if the cluster spin up time is caused by a large number of libraries getting installed during cluster startup time, take a look at <A href="https://docs.databricks.com/clusters/custom-containers.html" alt="https://docs.databricks.com/clusters/custom-containers.html" target="_blank">Databricks container services</A></P><P></P><P></P><P></P> Thu, 17 Jun 2021 19:58:07 GMT https://community.databricks.com/t5/data-engineering/any-recommendations-to-reduce-the-cluster-spin-up-time/m-p/23050#M15868 sajith_appukutt 2021-06-17T19:58:07Z https://community.databricks.com/t5/data-engineering/any-recommendations-to-reduce-the-cluster-spin-up-time/m-p/23051#M15869 <P>Databricks pool concept may help reduce cluster start up time. more details are <A href="https://docs.databricks.com/clusters/instance-pools/index.html" alt="https://docs.databricks.com/clusters/instance-pools/index.html" target="_blank">here</A></P> Thu, 17 Jun 2021 20:15:48 GMT https://community.databricks.com/t5/data-engineering/any-recommendations-to-reduce-the-cluster-spin-up-time/m-p/23051#M15869 Srikanth_Gupta_ 2021-06-17T20:15:48Z