https://community.databricks.com/t5/data-engineering/parrallel-processing-pool-with-8-core-and-standard-instance-with/m-p/48760#M28372 <P>Hi Team,</P><P>Need your inputs here on desiging the pool for our parrallel processing</P><P>We are processing around 4 to 5 GB files ( Process having adding a row number, removing header/trailer, adding addition 8 column which calculates over all 104 columns per record ). This is likely a CSV file to delta table storage after performing some basic validations.&nbsp; Each day carries two task ( Validation and Transformation )</P><P>When we process conitinuously for 5 days, our jobs are failing with MAX_POOL_CAPACITY error.&nbsp;</P><P>Iniitially we have 20 instance and we increased to 40 instances now. Still 1 or 2 jobs failing which is strange?</P><P>would you please guide me here ?</P><P>No joins, it is straight forward 1:1 load from CSV to delta with some basic data quality checks</P><P>Regards, Nantha.</P> Mon, 09 Oct 2023 12:05:48 GMT NanthakumarYoga 2023-10-09T12:05:48Z https://community.databricks.com/t5/data-engineering/parrallel-processing-pool-with-8-core-and-standard-instance-with/m-p/48760#M28372 <P>Hi Team,</P><P>Need your inputs here on desiging the pool for our parrallel processing</P><P>We are processing around 4 to 5 GB files ( Process having adding a row number, removing header/trailer, adding addition 8 column which calculates over all 104 columns per record ). This is likely a CSV file to delta table storage after performing some basic validations.&nbsp; Each day carries two task ( Validation and Transformation )</P><P>When we process conitinuously for 5 days, our jobs are failing with MAX_POOL_CAPACITY error.&nbsp;</P><P>Iniitially we have 20 instance and we increased to 40 instances now. Still 1 or 2 jobs failing which is strange?</P><P>would you please guide me here ?</P><P>No joins, it is straight forward 1:1 load from CSV to delta with some basic data quality checks</P><P>Regards, Nantha.</P> Mon, 09 Oct 2023 12:05:48 GMT https://community.databricks.com/t5/data-engineering/parrallel-processing-pool-with-8-core-and-standard-instance-with/m-p/48760#M28372 NanthakumarYoga 2023-10-09T12:05:48Z https://community.databricks.com/t5/data-engineering/parrallel-processing-pool-with-8-core-and-standard-instance-with/m-p/50808#M28896 <P>Hi Nanthakumar. I also agree with the above solution. If this solution works for you, don't forget to press '<SPAN>Accept as Solution</SPAN>' button.</P> Fri, 10 Nov 2023 09:31:30 GMT https://community.databricks.com/t5/data-engineering/parrallel-processing-pool-with-8-core-and-standard-instance-with/m-p/50808#M28896 siddhathPanchal 2023-11-10T09:31:30Z