topic Re: Need to extract data from delta tables and need to move it to on-prem, what's the best approach in Data Engineering

Need to extract data from delta tables and need to move it to on-prem, what's the best approach

Shivap — Wed, 12 Mar 2025 18:55:25 GMT

I want to extract data from databricks delta tables and need to move it to on-prem what's the best way to accomplish it -

Re: Need to extract data from delta tables and need to move it to on-prem, what's the best approach

Master_DataBric — Thu, 13 Mar 2025 02:55:56 GMT

You can extract the files and save them on cloud storage(s3/ADLS/GCS).
Then you can use the appropriate cloud CLI to read the files from cloud storage.
I think this is more scalable and can be achieved programmatically(i.e via your batch job).

Re: Need to extract data from delta tables and need to move it to on-prem, what's the best approach

sandeepmankikar — Thu, 13 Mar 2025 04:20:35 GMT

If you are planning to load data into an on-premises database, you can write a PySpark JDBC script to efficiently transfer data from Databricks Delta tables to the target database. This method enables seamless integration between Databricks and on-prem databases

Re: Need to extract data from delta tables and need to move it to on-prem, what's the best approach

Stefan-Koch — Thu, 13 Mar 2025 06:35:44 GMT

An easy way to do is to use Airbyte. You can run Airbyte locally, connect to databricks and copy the data to your OnPrem location.

https://docs.airbyte.com/integrations/destinations/databricks