https://community.databricks.com/t5/data-engineering/how-to-optimize-the-runtime-in-10-4-cluster/m-p/10920#M5963 <P>i am loading the 1billion data from spark dataframe into target table, but in the 7.3 cluster it takes 3 hours to complete but after migrated to 10.4 cluster its taking 8 hours to complete , how can i reduce the time duration​</P> Sun, 22 Jan 2023 12:58:10 GMT databicky 2023-01-22T12:58:10Z https://community.databricks.com/t5/data-engineering/how-to-optimize-the-runtime-in-10-4-cluster/m-p/10920#M5963 <P>i am loading the 1billion data from spark dataframe into target table, but in the 7.3 cluster it takes 3 hours to complete but after migrated to 10.4 cluster its taking 8 hours to complete , how can i reduce the time duration​</P> Sun, 22 Jan 2023 12:58:10 GMT https://community.databricks.com/t5/data-engineering/how-to-optimize-the-runtime-in-10-4-cluster/m-p/10920#M5963 databicky 2023-01-22T12:58:10Z https://community.databricks.com/t5/data-engineering/how-to-optimize-the-runtime-in-10-4-cluster/m-p/10922#M5965 <P>Hi @Mohammed sadamusean​,</P><P></P><P>Could you provide more details on what are you doing? What type of transformations/actions are you doing? whats your source and sink? batch or streaming? all that information will help.</P> Wed, 25 Jan 2023 01:33:35 GMT https://community.databricks.com/t5/data-engineering/how-to-optimize-the-runtime-in-10-4-cluster/m-p/10922#M5965 jose_gonzalez 2023-01-25T01:33:35Z https://community.databricks.com/t5/data-engineering/how-to-optimize-the-runtime-in-10-4-cluster/m-p/10923#M5966 <P>i have data in adls, i load thise data into multiple dataframes in the databricks notebook, from the final dataframe i am loading data into final target table based on the dataframe tempview, usually it takes 3 in 7.3 cluster but in 10.4 cluster it take around 8 hours , 1 billion records is there​</P> Wed, 25 Jan 2023 02:32:18 GMT https://community.databricks.com/t5/data-engineering/how-to-optimize-the-runtime-in-10-4-cluster/m-p/10923#M5966 databicky 2023-01-25T02:32:18Z https://community.databricks.com/t5/data-engineering/how-to-optimize-the-runtime-in-10-4-cluster/m-p/10924#M5967 <P>could you check your Spark UI to identify which stage is taking the longest time, and share some information in here</P> Fri, 24 Feb 2023 23:40:25 GMT https://community.databricks.com/t5/data-engineering/how-to-optimize-the-runtime-in-10-4-cluster/m-p/10924#M5967 jose_gonzalez 2023-02-24T23:40:25Z https://community.databricks.com/t5/data-engineering/how-to-optimize-the-runtime-in-10-4-cluster/m-p/10921#M5964 <P>Hi, Please refer <A href="https://docs.databricks.com/clusters/cluster-config-best-practices.html" target="test_blank">https://docs.databricks.com/clusters/cluster-config-best-practices.html</A> for best practises for cluster configurations. Please let us know if this helps. </P> Tue, 24 Jan 2023 08:06:56 GMT https://community.databricks.com/t5/data-engineering/how-to-optimize-the-runtime-in-10-4-cluster/m-p/10921#M5964 Debayan 2023-01-24T08:06:56Z