Dynamic Value References Not Working

[aseufert]

I can't get the dynamic value references to work in my jobs. I can use the deprecate references (e.g. job_id) but not the new references (e.g. job.id). As a test, I set a text widget called MyJobID following the example that will receive the dynamic value reference but cannot get the new references mentioned in the article to work.

This works and renders "my-job-123456":

{
  "MyJobID": "my-job-{{job_id}}"
}

While this does not and renders "my-job-{{job.id}}":

{
  "MyJobID": "my-job-{{job.id}}"
}

 I'm just using this as a simple test. What I'm really hoping to do is to use dynamic references to send task context from an upstream task dynamically, similar to the example of {{tasks.Get_user_data.values.name}} mentioned in the share task context article , but I can't even get a basic example working first.

Am I using these wrong? Why aren't the dynamic references getting interpolating into the widget?

[Shahfik]

Dynamic value reference doesn't work for me either. I had to use 

dbutils.jobs.taskValues.get(taskKey = "Get_user_data", key = "name") to retrieve the value

[themattmorris]

You're not alone, I'm experiencing the same. Reached out to Databricks support for help on this, will reply back if they help me find a resolution to the problem. Using the deprecated values for now.

[Kaniz]

Hi @aseufert, One possible reason for this issue could be that the new references are not supported in the version of Databricks you are using. You may want to check the Databricks documentation to ensure that the new references are supported in your version.

Another possible reason could be that the syntax of the new references is incorrect. You may want to check the syntax of the new references and ensure that they are correct.

Regarding your question about using dynamic references to send task context from an upstream task dynamically, you may want to refer to the Databricks documentation for more information on how to use dynamic references in Databricks.

If the issue persists, you may want to consider filing a support ticket to Databricks support for further assistance.

[themattmorris]

Thanks @Kaniz for the reply on reasons this may not be working. Regarding these potential problems, there is nothing in the documentation that suggests a minimum required version. But if you're aware of a minimum version requirement, do you mind sharing what that is? It would be helpful to have this in the documentation as well, if such a requirement exists. I've tried this on clusters with runtime version as late as 14.2 beta and have encountered the same problem on these as well.

Regarding a potential syntax error, the syntax in the original post follows what is provided in the documentation. But if you're aware of a potential syntax error in that code, do you mind commenting on that?

Thank you!

[aseufert]

Thanks for the reply. I'm using version webapp_2023-11-11_20.37.23Z_webappp_hotfix_20231106_ae45c0ec_1291847197

The jobs UI version is jobs-ui_2023-11-01_03.12.26Z_master_a08a2c30_1606137368

As @themattmorris said, I don't see the version mentioned anywhere in the documentation.

I also don't see the conditional task runs feature mentioned in the documentation so I assume I'm just on an old version of Databricks. Are you able to clarify which version of Databricks is required for this feature?

Thanks!

[themattmorris]

As I mentioned in an earlier post, I had been in contact with someone from Databricks regarding this problem. They got back to me yesterday indicating that Databricks actually still had to roll the feature out yesterday. This was done, and it is working now for me. I'd check again and see if these new reference values are working for you as well.

[aseufert]

@themattmorris Thanks for the update! I did see an alert that a new version of the job page was available. I was hoping it was the new features and sure enough, it was. I can confirm they're working for me as well.

Have a great day!

[themattmorris]

For what it's worth, it looks like job-level parameters were added with this update as well. I was wondering why I was unable to use those, but those are also working for me now.