Question on Ingestion Pipelines

[danny_frontgrad]

Is there a better way to select source tables than having to manually select them 1 by 1. I have 96 tables and it's a pain. The gui keeps back to the schema and i have to search through all the tables again. Is there a way to import the tables using the yaml file or json file?

[pradeep_singh]

Not sure about the exact ask here but if you are looking for metadata driven pipelines check dlt-meta

[pradeep_singh]

https://docs.databricks.com/aws/en/ldp/developer/dlt-meta

[danny_frontgrad]

I was in the ingestion pipeline. Under source tables i was asking if there was another to add source tables rather than adding them manually as in the picture attached. I have a couple of pipelines that have 96 source tables. This is the screen   

[danny_frontgrad]

I'm talking about here 

[pradeep_singh]

You can build the pipeline to ingest just 1 table first . then take the yaml configuration of the pipeline and edit it to add as many tables as you want . You can use a scripting logic to build this yaml for all the tables . Hope this helps 

[danny_frontgrad]

Thank you I read something about using the yaml file, It would be helpful to be more specific about steps on how to do this. Once I edit the yaml is it a way to import it back into the pipeline? Or am I running the yaml in a notebook. If you can answer this it would be helpful. Otherwise  i'll have to search for the steps online somewhere. Either way thank you for your time.