How to update external metastore cluster configuration on the fly ?

Oliver_Floyd — Thu, 03 Feb 2022 09:05:57 GMT

Hello,

In my use case, my data is pushed to an adls gen2 container called ingest

After some data processing on a databricks cluster of the ingest workspace, I declare the associated table in an external metastore for this workspace

At the end of this processing (according to certain criteria) I push the curated data (a simple copy) to other containers (lab/qal/prd, each container contains data for a databricks workspace)

and I want to declare the metastores for these 3 workspaces.

One solution is to launch 3 tasks after this first task. Each cluster associated with these tasks is configured with the metastore of each databricks workspace. It works but this solution is cumbersome:

need to start a cluster for each workspace
even if the table is already declared in the metastore, you have to start the cluster to check.
slow down our data process

Another solution could be to update the cluster configuration on the fly in the first task. I tried to modify the spark session configuration with the above lines of code:

spark.sparkContext.getConf().set("spark.hadoop.javax.jdo.option.ConnectionURL","jdbc:sqlserver://lab_env.database.windows.net:1433;database=labdatabase")

spark.conf.set("spark.hadoop.javax.jdo.option.ConnectionURL","jdbc:sqlserver://lab_env.database.windows.net:1433;database=labdatabase")

but it seems that it doesn't work.

My question is simple : Do you know if there is a way to change this configuration in a notebook, or if it is not possible at all

Thanking you in advance for your help

Re: How to update external metastore cluster configuration on the fly ?

Atanu — Sat, 12 Feb 2022 16:36:21 GMT

Hi @oliv vier as per our doc this can be achieved by only through

Spark config
Init script

So I think , on the fly it won't work. Thanks. But may be you can have this as feature request to our product team.

Re: How to update external metastore cluster configuration on the fly ?

Oliver_Floyd — Mon, 07 Mar 2022 08:16:27 GMT

Hello @Atanu Sarkar ,

Thank you for your answer. I have created a feature request. I hope, it will be soon accepted ^^

topic Re: How to update external metastore cluster configuration on the fly ? in Data Engineering

How to update external metastore cluster configuration on the fly ?

Re: How to update external metastore cluster configuration on the fly ?

Re: How to update external metastore cluster configuration on the fly ?