I want to create a function in azure Databricks notebook to send a email, based on a filter. Any leads are appriciated.

[ubsingh]

I have no idea from where to start

[lee · Accepted solution]

It is possible to use existing Python or Scala libraries to send an email from a Databricks notebook (e.g. python's smtplib). The answer in this forum question does a good job explaining further: https://forums.databricks.com/questions/17785/databricks-sendgrid-integration.htm This answer also talks about the 'jobs' feature of Databricks, which might fit your use case (i.e. configuring email rules for notebook runs)

If you are in an Azure environment and able to use other tools, I suggest looking at how (1) Data Factory, (2) Logic Apps, or (3) Functions might be used in your scenario. There are examples out there on Databricks and Azure sites if you do some searching. As mentioned above, it is possible to send emails from Databricks itself, but this is not the intended design of the platform. Instead, other tools in the landscape are better used for sending e-mails. The high level architecture might be something like

1. Data Factory job calls Databricks notebook
2. Databricks notebook returns information for filtering (i.e. some flag, or other metadata)
3. Data Factory job sends the email based on the output

[ubsingh]

Thanks for you help @leedabee. I will go through second option, First one is not applicable in my case.

[sshankar_satees]

@ubsingh - Just wondering how you managed doing this from databricks notebooks. I am trying to send a sample email to my or email via databricks python notebook. I tried to the sendbricks option however no luck with that. Not receiving emails. Any help would be appreciated. Its hard to believe its so twisted to achieve this simple email functionality in databricks. Earlier I was working on python with virtual environment on an edge node and this was pretty and straight and simple.