cancel
Showing results for 
Search instead for 
Did you mean: 
Data Engineering
Join discussions on data engineering best practices, architectures, and optimization strategies within the Databricks Community. Exchange insights and solutions with fellow data engineers.
cancel
Showing results for 
Search instead for 
Did you mean: 

I want to create a function in azure Databricks notebook to send a email, based on a filter. Any leads are appriciated.

ubsingh
New Contributor II

I have no idea from where to start

1 ACCEPTED SOLUTION

Accepted Solutions

lee
Contributor

It is possible to use existing Python or Scala libraries to send an email from a Databricks notebook (e.g. python's smtplib). The answer in this forum question does a good job explaining further: https://forums.databricks.com/questions/17785/databricks-sendgrid-integration.htm This answer also talks about the 'jobs' feature of Databricks, which might fit your use case (i.e. configuring email rules for notebook runs)

If you are in an Azure environment and able to use other tools, I suggest looking at how (1) Data Factory, (2) Logic Apps, or (3) Functions might be used in your scenario. There are examples out there on Databricks and Azure sites if you do some searching. As mentioned above, it is possible to send emails from Databricks itself, but this is not the intended design of the platform. Instead, other tools in the landscape are better used for sending e-mails. The high level architecture might be something like

  1. Data Factory job calls Databricks notebook
  2. Databricks notebook returns information for filtering (i.e. some flag, or other metadata)
  3. Data Factory job sends the email based on the output

View solution in original post

3 REPLIES 3

lee
Contributor

It is possible to use existing Python or Scala libraries to send an email from a Databricks notebook (e.g. python's smtplib). The answer in this forum question does a good job explaining further: https://forums.databricks.com/questions/17785/databricks-sendgrid-integration.htm This answer also talks about the 'jobs' feature of Databricks, which might fit your use case (i.e. configuring email rules for notebook runs)

If you are in an Azure environment and able to use other tools, I suggest looking at how (1) Data Factory, (2) Logic Apps, or (3) Functions might be used in your scenario. There are examples out there on Databricks and Azure sites if you do some searching. As mentioned above, it is possible to send emails from Databricks itself, but this is not the intended design of the platform. Instead, other tools in the landscape are better used for sending e-mails. The high level architecture might be something like

  1. Data Factory job calls Databricks notebook
  2. Databricks notebook returns information for filtering (i.e. some flag, or other metadata)
  3. Data Factory job sends the email based on the output

ubsingh
New Contributor II

Thanks for you help @leedabee. I will go through second option, First one is not applicable in my case.

@ubsingh - Just wondering how you managed doing this from databricks notebooks. I am trying to send a sample email to my or email via databricks python notebook. I tried to the sendbricks option however no luck with that. Not receiving emails. Any help would be appreciated. Its hard to believe its so twisted to achieve this simple email functionality in databricks. Earlier I was working on python with virtual environment on an edge node and this was pretty and straight and simple.

Join 100K+ Data Experts: Register Now & Grow with Us!

Excited to expand your horizons with us? Click here to Register and begin your journey to success!

Already a member? Login and join your local regional user group! If there isn’t one near you, fill out this form and we’ll create one for you to join!