How to identify S3 object type (directory or file) created by Databricks?
Options
- Mark as New
- Bookmark
- Subscribe
- Mute
- Subscribe to RSS Feed
- Permalink
- Report Inappropriate Content
08-07-2022 08:01 PM
The issue context is Delta Lake connector in Trino https://github.com/trinodb/trino/issues/13017
Trino identifies S3 object as a directory or a file using Content-Type header. Other query engines set application/x-directory in case of directories, but Databricks sets application/octet-stream in the header. Is there an alternative solution to identifies the object type using com.amazonaws.services.s3.model.ObjectMetadata when we use Databricks? If not, is it possible to request the header change?
Labels:
- Labels:
-
databricks
-
Delt Lake
-
Directory