shan_chandra
Databricks Employee
Databricks Employee

@dmart -  can you please try with a beefy driver and list out the root paths Eg. "/path/to/data1" , "/path/to/data2" ... and run the above code in multiple clusters by changing the root path to the above script. Please let us know if this works for you.