Need to extract data from delta tables and need to move it to on-prem, what's the best approach
- Mark as New
- Bookmark
- Subscribe
- Mute
- Subscribe to RSS Feed
- Permalink
- Report Inappropriate Content
03-12-2025 11:55 AM
I want to extract data from databricks delta tables and need to move it to on-prem what's the best way to accomplish it -
- Mark as New
- Bookmark
- Subscribe
- Mute
- Subscribe to RSS Feed
- Permalink
- Report Inappropriate Content
03-12-2025 07:55 PM
You can extract the files and save them on cloud storage(s3/ADLS/GCS).
Then you can use the appropriate cloud CLI to read the files from cloud storage.
I think this is more scalable and can be achieved programmatically(i.e via your batch job).
- Mark as New
- Bookmark
- Subscribe
- Mute
- Subscribe to RSS Feed
- Permalink
- Report Inappropriate Content
03-12-2025 09:20 PM
If you are planning to load data into an on-premises database, you can write a PySpark JDBC script to efficiently transfer data from Databricks Delta tables to the target database. This method enables seamless integration between Databricks and on-prem databases
- Mark as New
- Bookmark
- Subscribe
- Mute
- Subscribe to RSS Feed
- Permalink
- Report Inappropriate Content
03-12-2025 11:35 PM
An easy way to do is to use Airbyte. You can run Airbyte locally, connect to databricks and copy the data to your OnPrem location.
https://docs.airbyte.com/integrations/destinations/databricks
