Hello everyone,
We are currently facing performance issues when using Databricks as a transactional system with our .NET application via the Simba ODBC driver. Specifically, queries are taking 30 seconds to 1 minute to insert data into 8 to 10 tables, each having max to 400 records.
We also tested the Cdata ODBC and Cdata ADO.NET drivers for a proof of concept, but the results were not satisfactory. (https://www.cdata.com/drivers/).
Given the situation, we are considering switching to PostgreSQL as the transactional system for our .NET application, and then implementing a nightly data pipeline to incrementally load data from PostgreSQL into Databricks.
Before making this change, we would appreciate any suggestions on how we might improve performance with the current setup. Alternatively, do you think switching to PostgreSQL as the transactional system is the best approach?
Your expert opinions would be greatly appreciated. Thank you!
