01-17-2023 03:07 AM
Hello! Is there an equivalent of Create trigger on a table in Databricks sql?
CREATE TRIGGER [schema_name.]trigger_name
ON table_name
AFTER {[INSERT],[UPDATE],[DELETE]}
[NOT FOR REPLICATION]
AS
{sql_statements}
Thank you in advance!
01-18-2023 11:33 PM
You can try Auto Loader: Auto Loader supports two modes for detecting new files: directory listing and file notification.
Directory listing: Auto Loader identifies new files by listing the input directory. Directory listing mode allows you to quickly start Auto Loader streams without any permission configurations other than access to your data on cloud storage. In Databricks Runtime 9.1 and above, Auto Loader can automatically detect whether files are arriving with lexical ordering to your cloud storage and significantly reduce the amount of API calls it needs to make to detect new files.
File notification: Auto Loader can automatically set up a notification service and queue service that subscribe to file events from the input directory. File notification mode is more performant and scalable for large input directories or a high volume of files but requires additional cloud permissions for set up.
Refer - https://learn.microsoft.com/en-us/azure/databricks/spark/latest/structured-streaming/auto-loader
01-17-2023 04:35 AM
There currently isn't stored procedures or triggers.
01-17-2023 10:13 AM
Hi @ELENI GEORGOUSI(Customer) , We haven’t heard from you since the last response from @Joseph Kambourakis, and I was checking back to see if their suggestions helped you.
Or else, If you have any solution, please do share that with the community as it can be helpful to others.
Also, Please don't forget to click on the "Select As Best" button whenever the information provided helps resolve your question.
01-18-2023 11:33 PM
You can try Auto Loader: Auto Loader supports two modes for detecting new files: directory listing and file notification.
Directory listing: Auto Loader identifies new files by listing the input directory. Directory listing mode allows you to quickly start Auto Loader streams without any permission configurations other than access to your data on cloud storage. In Databricks Runtime 9.1 and above, Auto Loader can automatically detect whether files are arriving with lexical ordering to your cloud storage and significantly reduce the amount of API calls it needs to make to detect new files.
File notification: Auto Loader can automatically set up a notification service and queue service that subscribe to file events from the input directory. File notification mode is more performant and scalable for large input directories or a high volume of files but requires additional cloud permissions for set up.
Refer - https://learn.microsoft.com/en-us/azure/databricks/spark/latest/structured-streaming/auto-loader
Join our fast-growing data practitioner and expert community of 80K+ members, ready to discover, help and collaborate together while making meaningful connections.
Click here to register and join today!
Engage in exciting technical discussions, join a group with your peers and meet our Featured Members.