topic Re: Trouble accessing `_metadata` column using cloudFiles in Delta Live Tables in Data Engineering

Trouble accessing `_metadata` column using cloudFiles in Delta Live Tables

tej1 — Mon, 16 May 2022 17:07:45 GMT

We are building a delta live pipeline where we ingest csv files in AWS S3 using cloudFiles.

And it is necessary to access the file modification timestamp of the file.

As documented here, we tried selecting `_metadata` column in a task in delta live pipelines without success. Are we doing something wrong?

The code snippet is below:

@dlt.table(
    name = "bronze",
    comment = f"New {SCHEMA} data incrementally ingested from S3",
    table_properties = {
        "quality": "bronze"
    }
)
def bronze_job():
    return spark \
            .readStream \
            .format("cloudFiles") \
            .option("cloudFiles.useNotifications", "true") \
            .option("cloudFiles.format", "csv") \
            .option("cloudFiles.region", "eu-west-1") \
            .option("delimiter", ",") \
            .option("escape", "\"") \
            .option("header", "false") \
            .option("encoding", "UTF-8") \
            .schema(cdc_schema) \
            .load("/mnt/%s/cdc/%s" % (RAW_MOUNT_NAME, SCHEMA)) \
            .select("*", "_metadata")

Thanks.

Tejas

Re: Trouble accessing `_metadata` column using cloudFiles in Delta Live Tables

Hubert-Dudek — Tue, 17 May 2022 09:35:19 GMT

Are you using Databricks Runtime 10.5?

Re: Trouble accessing `_metadata` column using cloudFiles in Delta Live Tables

tej1 — Tue, 17 May 2022 12:42:54 GMT

Yes, on a standalone cluster (for any cluster outside of the DLT pipeline) this feature works using DR 10.5.

I found out the issue. We cannot choose run time (unable to set `spark_version`) in DLT pipeline settings. 😫

Re: Trouble accessing `_metadata` column using cloudFiles in Delta Live Tables

colt — Sat, 02 Jul 2022 18:42:22 GMT

I'm having the same problem. Does this answer mean that there is no way to get file metadata using Delta Live Tables?

Re: Trouble accessing `_metadata` column using cloudFiles in Delta Live Tables

Hubert-Dudek — Sun, 03 Jul 2022 18:13:52 GMT

Currently, DLT is running on runtime 10.3. Once it is 10.5 or higher, it should be possible.

Re: Trouble accessing `_metadata` column using cloudFiles in Delta Live Tables

tej1 — Wed, 03 Aug 2022 12:54:25 GMT

Update:

We were able to test `_metadata` column feature in DLT "preview" mode (which is DBR 11.0). Databricks doesn't recommend production workloads when using "preview" mode, but nevertheless, glad to be using this feature in DLT.