Simple integration to push data from third-party into a client's Databricks instance

[67]

Hi there, we have an industry data platform with multiple customers using it. We provide each customer with their own data every night via .csv. Some of our customers use Databricks, and import their data from us into it.

We would like to offer a more simple solution to remove the following steps: exporting a csv, uploading it to customer's S3, then customer importing it into their Databricks.

What is the best way we can partner with Databricks to integrate once and offer this benefit to any / all of our customers that use Databricks?

[jeffreyaven]

💡You could use external volumes with a Cloudflare R2 bucket as an intermediary - you write the nightly data files to R2 (using S3-compatible API), and your customers create external volumes in their Databricks workspace pointing to their designated R2 paths with read-only credentials you provide. This eliminates the manual CSV export/upload steps, has zero egress costs from R2, and doesn't require you to maintain your own Databricks infrastructure. Customers can then use Auto Loader or scheduled COPY INTO commands to automatically ingest new files as they arrive.