Lakehouse architectures seem enticing, especially from the standpoint of querying the data lake directly as it sits (as opposed to first migrating the data to an external data warehouse). While documentation and support seems pretty clear regarding support BI platforms like Tableau and Power BI, there is really no reference for using it as source for queries through custom APIs.
I'm wondering if anyone has recommendations for querying Databricks SQL from .NET Web APIs through ODBC connections? Is this even a supported or recommended use case? Would the recommendation for these types of queries be to execute them against more of a traditional data warehouse? If it's supported, what are the downsides of querying directly through Databricks SQL (Cost? Performance? Tooling?). I'm guessing I'd be somewhat limited since I'd need to install the Databricks ODBC driver in order to query it? Also, it seems there are limits to the number of concurret calls that can be made per cluster?
Many of these limitations make me think I should still push data to external data warehouses for these scenarios, but I wanted to see what everyone out there thinks.
Thanks!
Geoff
