https://community.databricks.com/t5/data-engineering/best-cluster-setup-for-intensive-transformation-workload/m-p/39401#M26961 <P>I have a pyspark dataframe, 61k rows, 3 columns, one of which is a string column which has a max length of 4k. I'm doing about 100 different regexp_replace functions on this dataframe, so, very resource intensive. I'm trying to write this to a delta table, but it seems no matter what compute I use I can't seem to get it to run in an hour. I know the code works because I limited it to 500 rows to test and it ran in about 30 seconds, so I know it just has to do with the magnitude of the data. Has anyone done something on this scale before, and do you know how I get this to run in an hour without breaking the bank?</P> Tue, 08 Aug 2023 18:35:12 GMT AChang 2023-08-08T18:35:12Z https://community.databricks.com/t5/data-engineering/best-cluster-setup-for-intensive-transformation-workload/m-p/39401#M26961 <P>I have a pyspark dataframe, 61k rows, 3 columns, one of which is a string column which has a max length of 4k. I'm doing about 100 different regexp_replace functions on this dataframe, so, very resource intensive. I'm trying to write this to a delta table, but it seems no matter what compute I use I can't seem to get it to run in an hour. I know the code works because I limited it to 500 rows to test and it ran in about 30 seconds, so I know it just has to do with the magnitude of the data. Has anyone done something on this scale before, and do you know how I get this to run in an hour without breaking the bank?</P> Tue, 08 Aug 2023 18:35:12 GMT https://community.databricks.com/t5/data-engineering/best-cluster-setup-for-intensive-transformation-workload/m-p/39401#M26961 AChang 2023-08-08T18:35:12Z https://community.databricks.com/t5/data-engineering/best-cluster-setup-for-intensive-transformation-workload/m-p/39447#M26983 <P>It seems that you're trying to apply a lot of transformations, but it's basic stuff, so I'd go for the best practices documentation and find a way to create a compute-optimized cluster.</P><P>Ref.:&nbsp;<A href="https://docs.databricks.com/en/clusters/cluster-config-best-practices.html#basic-batch-etl" target="_blank">https://docs.databricks.com/en/clusters/cluster-config-best-practices.html#basic-batch-etl</A></P> Wed, 09 Aug 2023 14:14:16 GMT https://community.databricks.com/t5/data-engineering/best-cluster-setup-for-intensive-transformation-workload/m-p/39447#M26983 Leonardo 2023-08-09T14:14:16Z