topic Re: Delta UserMetadata attribute using Serverless Compute in Data Engineering

Delta UserMetadata attribute using Serverless Compute

RobsonNLPT — Tue, 03 Sep 2024 10:11:54 GMT

To automate the configuration of Spark on serverless compute, Databricks has removed support for manually setting most Spark configurations.

I've used userMetadata attribute to add context for all workloads that write to delta tables

I have the follow options:

dataframe operations: opption"userMetadata","xxxxx")

spark.databricks.delta.commitInfo.userMetadata for scope level

SET spark.databricks.delta.commitInfo.userMetadata for sql

what about merge operations using dataframes using serverless compute ?

what about merge operations using dataframes or sql using serverless compute ?

deltaTable.as("t")

.merge(

changesDF.as("s"),

"s.PK=t.PK")

.whenMatched().updateAll

.whenNotMatched().insertAll

.execute()

Re: Delta UserMetadata attribute using Serverless Compute

VZLA — Mon, 23 Dec 2024 17:30:37 GMT

@RobsonNLPT

I see, so you're currently getting a "Configuration spark.databricks.delta.commitInfo.userMetadata is not available" correct?

I did an internal research and there doesn't seem to be a workaround for this yet, but it'll be good if you could raise such request directly from your workspace "Feedback" portal by clicking on the "?" next to your username on the top right.

Re: Delta UserMetadata attribute using Serverless Compute

RobsonNLPT — Mon, 23 Dec 2024 17:27:23 GMT

I've used that approach and it works if you use traditional clusters. In my case we are using serverless compute and those settings are not supported.