Spark Configurations with Serverless Compute

RobsonNLPT
Contributor III

I have some few problems to convert my notebooks run run with serverless compute.

Right now I can't set my delta userMetadata at session and  scope level using spark or sql.

Setting userMetadata at dataframe write operation is ok using the option: option("userMetadata","xxxxx")

but spark.databricks.delta.commitInfo.userMetadata for scope level and

set spark.databricks.delta.commitInfo.userMetadata for sql is not supported

Any plans to support or eliminate the limitation?

 

Best

 

 

 

  

 

Alberto_Umana
Databricks Employee
Databricks Employee

Hi @RobsonNLPT,

There is an internal feature request for this use-case. https://databricks.aha.io/ideas/ideas/DB-I-12401 and it's under idea and not ETA on its implementation yet.

I see.

I cant access  https://databricks.aha.io/ideas/ideas/DB-I-12401

Also I can't create my account