Delta UserMetadata attribute using Serverless Comp...

RobsonNLPT · ‎09-03-2024

To automate the configuration of Spark on serverless compute, Databricks has removed support for manually setting most Spark configurations.

I've used userMetadata attribute to add context for all workloads that write to delta tables

I have the follow options:

dataframe operations: opption"userMetadata","xxxxx")

spark.databricks.delta.commitInfo.userMetadata for scope level

SET spark.databricks.delta.commitInfo.userMetadata for sql

what about merge operations using dataframes using serverless compute ?

what about merge operations using dataframes or sql using serverless compute ?

deltaTable.as("t")

.merge(

changesDF.as("s"),

"s.PK=t.PK")

.whenMatched().updateAll

.whenNotMatched().insertAll

.execute()

VZLA · ‎12-23-2024

@RobsonNLPT

I see, so you're currently getting a "Configuration spark.databricks.delta.commitInfo.userMetadata is not available" correct?

I did an internal research and there doesn't seem to be a workaround for this yet, but it'll be good if you could raise such request directly from your workspace "Feedback" portal by clicking on the "?" next to your username on the top right.

RobsonNLPT · ‎12-23-2024

Hi

I've used that approach and it works if you use traditional clusters. In my case we are using serverless compute and those settings are not supported.

Delta UserMetadata attribute using Serverless Compute