Ryan_Chynoweth
Databricks Employee
Databricks Employee

Hi @baatch us​ , this is a great question. Option 1 is very ideal if you require realtime processing of your data. Since you noted that you only need to process data when you need I would think that Option 2 is a better choice for you.

Option 1 would require 24/7 processing (i.e. 24/7 cluster) which is more costly than you need. Since you can do batch processing Option 2 would be more cost effective. Event hubs should allow you to dump directly into ADLS without an intermediate tool.

If you ever do require stream processing it wouldn't be difficult to switch between your two options.

View solution in original post