topic Re: Asset Bundles define entire folder for source code transformation files in Data Engineering

Asset Bundles define entire folder for source code transformation files

susanne — Sat, 16 Aug 2025 13:04:41 GMT

Hi all 🙂

I used the new Lakeflow UI in order to create a pipeline. Now I am struggling with the asset bundle configuration.
When I am creating the pipeline manually I can configure the correct folder to the transformations where my sql and python transformation files are stored like this:
But I dont know how to configure this via asset bundles.
Right now I have this code in my resources/pipeline.yml

resources: pipelines: cdc_cdr_demo_pipeline: name: cdc_cdr_demo_pipeline libraries: - file: path: /Workspace/Users/[my-user-name]/.bundle/demo_cdc_cdr_pipeline/dev/files/transformations/ serverless: true photon: true catalog: ${var.catalog} schema: ${var.schema} development: true channel: "PREVIEW"

and this in my databricks.yml :

bundle: name: demo_cdc_cdr_pipeline_databricks_bundle include: - resources/*.yml variables: catalog: description: The catalog to use schema: description: The schema to use notifications: description: The email addresses to use for failure notifications targets: dev: # The default target uses 'mode: development' to create a development copy. # - Deployed resources get prefixed with '[dev my_user_name]' # - Any job schedules and triggers are paused by default. # See also https://docs.databricks.com/dev-tools/bundles/deployment-modes.html. mode: development default: true workspace: host: host-url root_path: /Workspace/Users/${workspace.current_user.userName}/.bundle/demo_cdc_cdr_pipeline/dev variables: # env: "dev_${workspace.current_user.short_name}_" catalog: sandbox schema: user_name notifications: [] prod: mode: production workspace: host: host-url variables: catalog: sandbox_dui_prod schema: user_name notifications: []

What is the correct asset bundle configuration to set all files in the transformations folder as source code in the new Lakeflow?

Thanks a lot in advance and best regards,
Susanne

Re: Asset Bundles define entire folder for source code transformation files

szymon_dybczak — Sat, 16 Aug 2025 14:01:34 GMT

Hi @susanne

When you click YAML button it should return to you proper definiton of pipeline configured via UI that you can use in DAB.

But basically, you're looking for libraries mapping (in UI called source code)

id: 142be57c-6c0f-435c-8f4f-4ba45fdb3992 pipeline_type: WORKSPACE name: test libraries: - glob: include: /Workspace/Users/sdybczak2382@gmail.com/test/transformations/** schema: default continuous: false development: false photon: true channel: CURRENT catalog: workspace serverless: true root_path: /Workspace/Users/sdybczak2382@gmail.com/test

Re: Asset Bundles define entire folder for source code transformation files

susanne — Sat, 16 Aug 2025 15:53:18 GMT

Hi Szymon,
thanks once again for your help!
It worked now with your approach.

Do you maybe know why there is this warning displayed after databricks bundle validate/deploy:
Warning: unknown field: glob

This was one reason I thought this can not be the right configuration.

Have a nice day,
Susanne

Re: Asset Bundles define entire folder for source code transformation files

szymon_dybczak — Sat, 16 Aug 2025 16:00:41 GMT

Hi @susanne ,

Cool, glad that it worked for you. Regarding warning, maybe you have outdated version of databricks cli?