topic Re: Question on Ingestion Pipelines in Data Engineering

Question on Ingestion Pipelines

danny_frontgrad — Sat, 24 Jan 2026 18:08:04 GMT

Is there a better way to select source tables than having to manually select them 1 by 1. I have 96 tables and it's a pain. The gui keeps back to the schema and i have to search through all the tables again. Is there a way to import the tables using the yaml file or json file?

Re: Question on Ingestion Pipelines

pradeep_singh — Sat, 24 Jan 2026 22:53:57 GMT

Not sure about the exact ask here but if you are looking for metadata driven pipelines check dlt-meta

Re: Question on Ingestion Pipelines

pradeep_singh — Sat, 24 Jan 2026 22:55:14 GMT

https://docs.databricks.com/aws/en/ldp/developer/dlt-meta

Re: Question on Ingestion Pipelines

danny_frontgrad — Sat, 24 Jan 2026 23:04:58 GMT

I'm talking about here

Re: Question on Ingestion Pipelines

danny_frontgrad — Sun, 25 Jan 2026 00:44:56 GMT

I was in the ingestion pipeline. Under source tables i was asking if there was another to add source tables rather than adding them manually as in the picture attached. I have a couple of pipelines that have 96 source tables. This is the screen

Re: Question on Ingestion Pipelines

pradeep_singh — Sun, 25 Jan 2026 01:32:22 GMT

You can build the pipeline to ingest just 1 table first . then take the yaml configuration of the pipeline and edit it to add as many tables as you want . You can use a scripting logic to build this yaml for all the tables . Hope this helps

Re: Question on Ingestion Pipelines

danny_frontgrad — Sun, 25 Jan 2026 01:48:38 GMT

Thank you I read something about using the yaml file, It would be helpful to be more specific about steps on how to do this. Once I edit the yaml is it a way to import it back into the pipeline? Or am I running the yaml in a notebook. If you can answer this it would be helpful. Otherwise i'll have to search for the steps online somewhere. Either way thank you for your time.

Re: Question on Ingestion Pipelines

pradeep_singh — Sun, 25 Jan 2026 02:35:16 GMT

1 - Top right corner . Click on edit pipeline

2 - On the next screen again on top right corner click the 3 vertical dots to find Switch to code version (YAML)

3 - Edit this yaml file .

If you want to import all the tables from a specific sql server schema use objects as schema . example .

If you want to import specific tables use objects as table . example below . you can edit this yaml to add the tables manually or using a script .

Re: Question on Ingestion Pipelines

danny_frontgrad — Sun, 25 Jan 2026 02:42:43 GMT

In my version I can't edit the yaml file just view it in the UI. so not sure where to put the edited yaml in the UI for it to take effect. Is it done through Databricks CLI. But appreciate your help

Re: Question on Ingestion Pipelines

pradeep_singh — Sun, 25 Jan 2026 03:25:30 GMT

So you dont see the option to edit the pipeline ?

once you click on edit pipeline you dont see the option to Switch to code version(YAML)

After you Switch to code version(YAML) you can only view that yaml and cant edit it ?

Re: Question on Ingestion Pipelines

danny_frontgrad — Sun, 25 Jan 2026 03:52:03 GMT

Sorry I was on the wrong screen, i do see the option to Switch to code version (YAML) so I can edit from there and save got you thank you

Re: Question on Ingestion Pipelines

pradeep_singh — Sun, 25 Jan 2026 04:04:07 GMT

Cheers !!