how can I create a job in databricks with parameters of isolated from synapses or azure datafactory, because I can not find any option that allows to pass as parameter this value and not being able to do so I have no access to my unit catalog in databricks
example:
{
"num_workers": 1,
"cluster_name": "...",
"spark_version": "14.0.x-scala2.12",
"spark_conf": {
"spark.hadoop.fs.azure.account.oauth2.client.endpoint": "...",
"spark.hadoop.fs.azure.account.auth.type": "...",
"spark.hadoop.fs.azure.account.oauth.provider.type": "...",
"spark.hadoop.fs.azure.account.oauth2.client.id": "...",
"spark.hadoop.fs.azure.account.oauth2.client.secret": "..."
},
"node_type_id": "...",
"driver_node_type_id": "...",
"ssh_public_keys": [],
"spark_env_vars": {
"cluster_type": "all-purpose"
},
"init_scripts": [],
"enable_local_disk_encryption": false,
"data_security_mode": "USER_ISOLATION",
"cluster_id": "..."
}
Hi warner thanks for your question, I share the link service in synapses updated, currently we have a pool in databricks then what we do with the link service is that it creates a job and uploads an instance with the resources of our pool but to upload the cluster we have not identified where to send the parameter of isolated, because we need it to see our unit catalog, then in our flow we call the notebook using this link service, and in our code we make use of unit catalog
these are the characteristics with which the cluster is created, but these are the ones that I need
I hope this helps to understand a little, and thank you for your help
