I see some issues discussed in the airflow like this but it was fixed in the later versions. Try upgrading the provider version and let us know if it fixes the issue.
Hi @15460 , are we sure this issue is related to Idempotency token? The error you are encountering may not be related to the Idempotency token. What is your airflow provider version? Can you please upgrade to the latest version and retry?
@Sulfikkar Basheer Shylaja​ Is your cluster using a custom docker image? The problem here is, the local filesystem path (file:/) is only supported if you use a custom docker image. In regular DBR runtimes, the init script can only be pulled from dbfs...
Hi @Sulfikkar Basheer Shylaja​ , Why don't you store the init-script on DBFS and just pass the dbfs:/ path of the init script in Pulumi? You could just run this code on a notebook-%python
dbutils.fs.put("/databricks/init-scripts/set-private-pip-repos...
Hi @data engineer​ can you elaborate on this issue? If you are running Python UDF on a shared access mode cluster, it might not work. Can you try this on a personal compute cluster/single-user access mode cluster?
