Hi @tyhatwar785 

1. Should metadata and file download be separate jobs/notebooks or combined?
Keep them in separate notebooks but orchestrate them under a single Databricks Job.
for better error handling, and retries .

2. Cluster recommendations
start with a general-purpose cluster( Standard_DS4_v2 (28 GB memory, 8 vCPU) ) with autoscaling enabled

3. Parallelism
If all processing is inside Databricks

4. Best practices

Retries: Use Databricks Job-level retries and add custom retry logic using UDF

Error Handling: Use Python’s try/except with structured logging (logging library) for better observability.

Monitoring: Integrate with Databricks Lakehouse Monitoring or send metrics/logs