BilalAslamDbrx
Databricks Employee
Databricks Employee

Swetha, why not use Workflows? There's a powerful API for getting all the metadata you need. An no need to write code to manage parallelism yourself.

View solution in original post