Question on Ingestion Pipelines

[danny_frontgrad]

Is there a better way to select source tables than having to manually select them 1 by 1. I have 96 tables and it's a pain. The gui keeps back to the schema and i have to search through all the tables again. Is there a way to import the tables using the yaml file or json file?

[pradeep_singh]

Not sure about the exact ask here but if you are looking for metadata driven pipelines check dlt-meta

[pradeep_singh]

https://docs.databricks.com/aws/en/ldp/developer/dlt-meta

[danny_frontgrad]

I was in the ingestion pipeline. Under source tables i was asking if there was another to add source tables rather than adding them manually as in the picture attached. I have a couple of pipelines that have 96 source tables. This is the screen   

[danny_frontgrad]

I'm talking about here 

[pradeep_singh]

You can build the pipeline to ingest just 1 table first . then take the yaml configuration of the pipeline and edit it to add as many tables as you want . You can use a scripting logic to build this yaml for all the tables . Hope this helps 

[danny_frontgrad]

Thank you I read something about using the yaml file, It would be helpful to be more specific about steps on how to do this. Once I edit the yaml is it a way to import it back into the pipeline? Or am I running the yaml in a notebook. If you can answer this it would be helpful. Otherwise  i'll have to search for the steps online somewhere. Either way thank you for your time. 

[pradeep_singh · Accepted solution]

1 - Top right corner . Click on edit pipeline 

2 - On the next screen again on top right corner click the 3 vertical dots to find Switch to code version (YAML) 

3 - Edit this yaml file . 

If you want to import all the tables from a specific sql server schema use objects as schema . example .

If you want to import specific tables use objects as table . example below . you can edit this yaml to add the tables manually or using a script . 

[danny_frontgrad]

In my version I can't edit the yaml file just view it in the UI. so not sure where to put the edited yaml in the UI for it to take effect. Is it done through Databricks CLI. But appreciate your help