spark config not working in job cluster

thot — Tue, 10 Sep 2024 03:04:32 GMT

I am trying to rename a delta table like this:

spark.conf.set("spark.databricks.delta.alterTable.rename.enabledOnAWS", "true")

spark.sql("ALTER TABLE db1.rz_test5 RENAME TO db1.rz_test6")

The data is on aws s3, that's why I have to use spark config in the notebook. This works fine in the interactive cluster.

But when I test it in a job using a job cluster it failed with error:

org.apache.hadoop.hive.ql.metadata.HiveException: Unable to alter table.

The runtime for the interactive cluster is: 13.3 LTS

no spark configuration in the cluster.

The runtime for the job cluster in the job is: 14.3 LTS

no spark configuration in the cluster.

Both clusters use the same instance profile.

Re: spark config not working in job cluster

MaximeGendre — Tue, 10 Sep 2024 06:44:25 GMT

Hello,
it would be interesting to test with the same runtime version.
Does it work with a job running on a 13.3 ?

thot — Tue, 10 Sep 2024 07:17:49 GMT

No, 13.3 not working and has the same result.