Anonymous
Not applicable

You have a couple of options to consider:

  1. External Location: You can create an external location in your Unity metastore that points to the landing bucket in S3. This allows Unity to access the files in that location without having to copy or move them to the default location managed by Unity. You can configure the external location using the Unity Catalog's administration tools or by using the Unity SDK/API.

    To create the external location, specify the S3 bucket and prefix (folder) where the files are located. Unity will be able to read the data directly from the specified S3 location without any data movement.

  2. Direct Read: Unity also provides the ability to directly read data from files in S3 without the need for an external location. In this approach, you can directly query the files in the S3 landing bucket using SQL or Spark commands. Unity will use its underlying query engine to perform distributed processing and retrieve the data from the S3 files.

    To read the data directly from the S3 landing bucket, you can use the Unity Catalog's SQL or Spark interfaces to interact with the data and perform the necessary operations like filtering, aggregating, or joining the datasets.