topic Re: Spark Configurations with Serverless Compute in Data Engineering

Spark Configurations with Serverless Compute

RobsonNLPT — Wed, 29 Jan 2025 14:09:05 GMT

I have some few problems to convert my notebooks run run with serverless compute.

Right now I can't set my delta userMetadata at session and scope level using spark or sql.

Setting userMetadata at dataframe write operation is ok using the option: option("userMetadata","xxxxx")

but spark.databricks.delta.commitInfo.userMetadata for scope level and

set spark.databricks.delta.commitInfo.userMetadata for sql is not supported

Any plans to support or eliminate the limitation?

Best

Alberto_Umana — Wed, 29 Jan 2025 15:30:51 GMT

There is an internal feature request for this use-case. https://databricks.aha.io/ideas/ideas/DB-I-12401 and it's under idea and not ETA on its implementation yet.

RobsonNLPT — Wed, 29 Jan 2025 15:57:01 GMT

I see.

Also I can't create my account