We currently have approximately 60 tables that collectively exceed 500GB, representing 99.9% of our total database size. One potential solution is to migrate these larger tables to Databricks, which may help us mitigate synchronization costs and address various performance challenges associated with SQL Server.
In Rails, we have the capability to define distinct connections for different models. This would allow us to redirect the larger tables to Databricks while maintaining normal functionality for the rest of the application. If we have a compatible Active Record adapter, it should integrate with minimal adjustments, albeit potentially at a slightly reduced performance compared to SQL.
If you have worked on this use case or have any insights, I would greatly appreciate your input.
Thank you for your help!
