How to add webhook notification in DLT pipeline through yml

[guptaharsh]

Hi team,

I am trying to create a slack webhook notification in a DLT pipeline leveraging jobs.yml. 

targets:

  dev:

    # The default target uses 'mode: development' to create a development copy.

    # - Deployed resources get prefixed with '[dev my_user_name]'

    # - Any job schedules and triggers are paused by default.

    # See also https://docs.databricks.com/dev-tools/bundles/deployment-modes.html.

    mode: development

    default: true

    workspace:

      host: https://dbc-gh27.cloud.databricks.com

    variables:

      catalog: data

      schema: sap_data

      notifications: email@jkmail.com]

      webhook: https://hooks.slack.com/services6CdQmlT

I am trying to save this yml and deploy but the slack webhook its not working.

#DLT #pipeline

[szymon_dybczak · Accepted solution]

Hi @guptaharsh ,

Out of the box, there's currently no way to configure notification via webhook. Only email is supported as a notification medium for pipeline. Check yourself at rest api docs:

Create a pipeline | Pipelines API | REST API reference | Databricks on AWS

But you can use custom monitoring to achieve what you want. Here's an example from documentation where they sending event to a Slack channel using webhook

Define custom monitoring of Lakeflow Declarative Pipelines with event hooks | Databricks on AWS

[guptaharsh]

@szymon_dybczak , thanks for the update. I thought like as we can add slack notification in jobs.

[szymon_dybczak]

No problem @guptaharsh  🙂 And if the answer was helpful to you consider marking it as a solution to this thread

[guptaharsh · Accepted solution]

@szymon_dybczak , I gone through the Databricks docs.. I have added the webhook notifications in DLT pipeline. 
it works fine for me 😊.

# The job that triggers api_data_pipeline.

resources:

  jobs:

    api_data_job:

      name: api_data_job

      schedule:

        quartz_cron_expression: 0 30 5 * * ?

        timezone_id: Asia/Kolkata

        pause_status: UNPAUSED

      webhook_notifications:

        on_failure:

          - id: ${var.webhook}

        on_success:

          - id: ${var.webhook}

 

[szymon_dybczak]

That's awesome @guptaharsh , thanks for sharing!