Can Databricks write query results to s3 in another account via the API
- Mark as New
- Bookmark
- Subscribe
- Mute
- Subscribe to RSS Feed
- Permalink
- Report Inappropriate Content
08-22-2024 08:34 AM
I work for a company where we are trying to create a Databrick's integration in node using the @DataBricks/sql package to query customers clusters or warehouses. I see documentation of being able to load data via a query from s3 using STS tokens where you do something like:
COPY INTO my_json_data FROM 's3://my-bucket/jsonData' WITH ( CREDENTIAL (AWS_ACCESS_KEY = '...', AWS_SECRET_KEY = '...', AWS_SESSION_TOKEN = '...') ) FILEFORMAT = JSON
but the inverse doesn't seem to be true where you can use these to copy query results directly into s3 like doing something like
COPY INTO 's3://my-bucket/jsonData'
FROM (SELECT * my_json_data) WITH
( CREDENTIAL (AWS_ACCESS_KEY = '...', AWS_SECRET_KEY = '...', AWS_SESSION_TOKEN = '...') )
FILEFORMAT = JSON
Does anyone know if capabilities like that exist? We'd like to be able to run a query and have the customer's Databrick's instance write results to an S3 bucket in our account.
- Labels:
-
Workflows
- Mark as New
- Bookmark
- Subscribe
- Mute
- Subscribe to RSS Feed
- Permalink
- Report Inappropriate Content
12-15-2024 07:13 PM
Have you been able to get a response on this topic, based on the information I can see it might not be supported to write on an S3 outside your account