Trouble accessing `_metadata` column using cloudFiles in Delta Live Tables

[tej1]

We are building a delta live pipeline where we ingest csv files in AWS S3 using cloudFiles.

And it is necessary to access the file modification timestamp of the file.

As documented here, we tried selecting `_metadata` column in a task in delta live pipelines without success. Are we doing something wrong?

The code snippet is below:

@dlt.table(
    name = "bronze",
    comment = f"New {SCHEMA} data incrementally ingested from S3",
    table_properties = {
        "quality": "bronze"
    }
)
def bronze_job():
    return spark \
            .readStream \
            .format("cloudFiles") \
            .option("cloudFiles.useNotifications", "true") \
            .option("cloudFiles.format", "csv") \
            .option("cloudFiles.region", "eu-west-1") \
            .option("delimiter", ",") \
            .option("escape", "\"") \
            .option("header", "false") \
            .option("encoding", "UTF-8") \
            .schema(cdc_schema) \
            .load("/mnt/%s/cdc/%s" % (RAW_MOUNT_NAME, SCHEMA)) \
            .select("*", "_metadata")

Thanks.

Tejas

[Hubert-Dudek]

Are you using Databricks Runtime 10.5?

[tej1 · Accepted solution]

Yes, on a standalone cluster (for any cluster outside of the DLT pipeline) this feature works using DR 10.5.

I found out the issue. We cannot choose run time (unable to set `spark_version`) in DLT pipeline settings. 😫

[colt]

I'm having the same problem. Does this answer mean that there is no way to get file metadata using Delta Live Tables?

[Hubert-Dudek]

Currently, DLT is running on runtime 10.3. Once it is 10.5 or higher, it should be possible.

[tej1]

Update:

We were able to test `_metadata` column feature in DLT "preview" mode (which is DBR 11.0). Databricks doesn't recommend production workloads when using "preview" mode, but nevertheless, glad to be using this feature in DLT.