Hello @eballinger , thank you for your question. To better assist you, could you clarify a few details?

• Are you seeing delays in specific stages (e.g., metadata fetching, schema validation, or table setup)?
• Could you provide more details on how the dynamic declaration is implemented (e.g., looping structure or table metadata source)?
• Have you profiled the pipeline to identify which part of the initialization is taking longer?

The increased runtime with dynamic declarations is likely due to the overhead of processing each table dynamically, compared to the static approach where these computations are predefined. To address this:

• Batch Processing: Process tables in smaller batches instead of handling all 300+ tables in one go.
• Parallel Execution: Explore parallel or asynchronous processing for table declarations.
• Metadata Optimization: Cache reusable metadata (e.g., schemas or paths) to minimize repeated operations in the loop.
• External Configuration: Use a configuration file (e.g., JSON or YAML) for table definitions to simplify and speed up initialization.

If these don’t resolve the issue, let me know more specifics, and I’ll provide further suggestions!