https://community.databricks.com/t5/data-engineering/cloning-the-data-between-two-catalogs/m-p/100322#M40266 <P>Hello community,</P><P>I was writing this piece of code to do the data migration between two catalogs:<BR /><BR /></P><P>&nbsp;</P><LI-CODE lang="markup"> # Read data and partitioning print(f"Loading {table_name} from production catalog...") prod_df_table_name = f"prod_catalog.`00_bronze_layer`.pg_{table_name}" prod_df_table = spark.read.table(prod_df_table_name).persist(StorageLevel.MEMORY_AND_DISK) # Write with optimizations print(f"Saving {table_name} to Bronze Schema") stg_df_table_name = f"stg_catalog.`00_bronze_layer`.pg_{table_name}" if partition_col in ["all", "none", "export_run_timestamp", "updated_at", "received_at"]: prod_df_table.write \ .format("delta") \ .mode("overwrite") \ .option("mergeSchema", "true") \ .saveAsTable(stg_df_table_name) else: prod_df_table.write \ .format("delta") \ .mode("overwrite") \ .partitionBy(partition_col) \ .option("mergeSchema", "true") \ .saveAsTable(stg_df_table_name) prod_df_table.unpersist()</LI-CODE><P>&nbsp;</P><P>I also using a little cluster with 14 gb and 4 cores without any worker node, any suggestion to improve the speed of copy :(? I'm going to work with tables that have sometimes 10 gigabytes of data</P> Thu, 28 Nov 2024 11:17:55 GMT jeremy98 2024-11-28T11:17:55Z https://community.databricks.com/t5/data-engineering/cloning-the-data-between-two-catalogs/m-p/100322#M40266 <P>Hello community,</P><P>I was writing this piece of code to do the data migration between two catalogs:<BR /><BR /></P><P>&nbsp;</P><LI-CODE lang="markup"> # Read data and partitioning print(f"Loading {table_name} from production catalog...") prod_df_table_name = f"prod_catalog.`00_bronze_layer`.pg_{table_name}" prod_df_table = spark.read.table(prod_df_table_name).persist(StorageLevel.MEMORY_AND_DISK) # Write with optimizations print(f"Saving {table_name} to Bronze Schema") stg_df_table_name = f"stg_catalog.`00_bronze_layer`.pg_{table_name}" if partition_col in ["all", "none", "export_run_timestamp", "updated_at", "received_at"]: prod_df_table.write \ .format("delta") \ .mode("overwrite") \ .option("mergeSchema", "true") \ .saveAsTable(stg_df_table_name) else: prod_df_table.write \ .format("delta") \ .mode("overwrite") \ .partitionBy(partition_col) \ .option("mergeSchema", "true") \ .saveAsTable(stg_df_table_name) prod_df_table.unpersist()</LI-CODE><P>&nbsp;</P><P>I also using a little cluster with 14 gb and 4 cores without any worker node, any suggestion to improve the speed of copy :(? I'm going to work with tables that have sometimes 10 gigabytes of data</P> Thu, 28 Nov 2024 11:17:55 GMT https://community.databricks.com/t5/data-engineering/cloning-the-data-between-two-catalogs/m-p/100322#M40266 jeremy98 2024-11-28T11:17:55Z https://community.databricks.com/t5/data-engineering/cloning-the-data-between-two-catalogs/m-p/100339#M40267 <P>FYI,<BR />I did it increasing the size of the cluster using much cores and directly written:<BR /><BR /></P><DIV><DIV><SPAN>prod_df_table.write \</SPAN></DIV><DIV><SPAN>.</SPAN><SPAN>format</SPAN><SPAN>(</SPAN><SPAN>"delta"</SPAN><SPAN>) \</SPAN></DIV><DIV><SPAN>.</SPAN><SPAN>mode</SPAN><SPAN>(</SPAN><SPAN>"overwrite"</SPAN><SPAN>) \</SPAN></DIV><DIV><SPAN>.</SPAN><SPAN>saveAsTable</SPAN><SPAN>(stg_df_table_name)</SPAN></DIV></DIV> Thu, 28 Nov 2024 13:47:15 GMT https://community.databricks.com/t5/data-engineering/cloning-the-data-between-two-catalogs/m-p/100339#M40267 jeremy98 2024-11-28T13:47:15Z