Hi Phani, what exactly are you looking for with best practices?
At a high level:
- Always provide an external storage location (S3, ADLS, GCS) for your pipeline
- Use Auto Scaling!
- Python imports can be leverage to reuse code
With regards to providing a storage location, if you put all your pipelines in a common storage location it makes it easier to read all the associated event logs for pipeline monitoring as well.