@elkaganeva,

When you register an S3 bucket as an external location in Unity Catalog, you can directly access Delta tables stored in that bucket using the spark.readStream and spark.writeStream methods. The metadata for the Delta tables is managed through Unity Catalog, not stored in the bucket itself. This means that Unity Catalog handles the metadata management, ensuring that access control and other governance policies are enforced.

Here is the code snippet you provided, which will work as expected:

spark.readStream

  .format("delta")

  .load("s3://my-bucket/delta-table-1/")

  .writeStream

  .format("delta")

  .option("checkpointLocation", aggregatedEvidencesCheckpoint)

  .trigger(Trigger.AvailableNow())

  .start("s3://my-bucket/delta-table-2/")