Asset Bundles define entire folder for source code transformation files

[susanne]

Hi all 🙂

I used the new Lakeflow UI in order to create a pipeline. Now I am struggling with the asset bundle configuration. 
When I am creating the pipeline manually I can configure the correct folder to the transformations where my sql and python transformation files are stored like this: 
But I dont know how to configure this via asset bundles.
Right now I have this code in my resources/pipeline.yml

resources:
  pipelines:
    cdc_cdr_demo_pipeline:
      name: cdc_cdr_demo_pipeline

      libraries:
        - file: 
            path: /Workspace/Users/[my-user-name]/.bundle/demo_cdc_cdr_pipeline/dev/files/transformations/
      serverless: true 
      photon: true
      catalog: ${var.catalog}
      schema: ${var.schema}
      development: true
      channel: "PREVIEW"

and this in my databricks.yml :

bundle:
  name: demo_cdc_cdr_pipeline_databricks_bundle

include: 
  - resources/*.yml

variables:
  catalog:
    description: The catalog to use
  schema:
    description: The schema to use
  notifications:
    description: The email addresses to use for failure notifications

targets:
  dev:
    # The default target uses 'mode: development' to create a development copy.
    # - Deployed resources get prefixed with '[dev my_user_name]'
    # - Any job schedules and triggers are paused by default.
    # See also https://docs.databricks.com/dev-tools/bundles/deployment-modes.html.
    mode: development
    default: true
    workspace:
      host: host-url
      root_path: /Workspace/Users/${workspace.current_user.userName}/.bundle/demo_cdc_cdr_pipeline/dev
    variables:
    #  env: "dev_${workspace.current_user.short_name}_"
      catalog: sandbox
      schema: user_name
      notifications: []

  prod:
    mode: production
    workspace:
      host: host-url
    variables:
      catalog: sandbox_dui_prod
      schema: user_name
      notifications: []
        

What is the correct asset bundle configuration to set all files in the transformations folder as source code in the new Lakeflow?

Thanks a lot in advance and best regards, 
Susanne

[szymon_dybczak · Accepted solution]

Hi @susanne 

When you click YAML button it should return to you proper definiton of pipeline configured via UI that you can use in DAB.

But basically, you're looking for libraries mapping (in UI called source code)

id: 142be57c-6c0f-435c-8f4f-4ba45fdb3992
pipeline_type: WORKSPACE
name: test
libraries:
  - glob:
      include: /Workspace/Users/sdybczak2382@gmail.com/test/transformations/**
schema: default
continuous: false
development: false
photon: true
channel: CURRENT
catalog: workspace
serverless: true
root_path: /Workspace/Users/sdybczak2382@gmail.com/test

[susanne]

Hi Szymon, 
thanks once again for your help!
It worked now with your approach. 

Do you maybe know why there is this warning displayed after databricks bundle validate/deploy:
Warning: unknown field: glob

This was one reason I thought this can not be the right configuration.

Have a nice day,
Susanne

[szymon_dybczak]

Hi @susanne ,

Cool, glad that it worked for you. Regarding warning, maybe you have outdated version of databricks cli?