topic Re: Dynamic Value References Not Working in Data Engineering

Dynamic Value References Not Working

aseufert — Tue, 31 Oct 2023 16:36:04 GMT

I can't get the dynamic value references to work in my jobs. I can use the deprecate references (e.g. job_id) but not the new references (e.g. job.id). As a test, I set a text widget called MyJobID following the example that will receive the dynamic value reference but cannot get the new references mentioned in the article to work.

This works and renders "my-job-123456":

{
  "MyJobID": "my-job-{{job_id}}"
}

While this does not and renders "my-job-{{job.id}}":

{
  "MyJobID": "my-job-{{job.id}}"
}

I'm just using this as a simple test. What I'm really hoping to do is to use dynamic references to send task context from an upstream task dynamically, similar to the example of {{tasks.Get_user_data.values.name}} mentioned in the share task context article , but I can't even get a basic example working first.

Am I using these wrong? Why aren't the dynamic references getting interpolating into the widget?

Re: Dynamic Value References Not Working

Shahfik — Fri, 10 Nov 2023 13:15:48 GMT

Dynamic value reference doesn't work for me either. I had to use

dbutils.jobs.taskValues.get(taskKey = "Get_user_data", key = "name") to retrieve the value

Re: Dynamic Value References Not Working

themattmorris — Tue, 14 Nov 2023 01:34:58 GMT

You're not alone, I'm experiencing the same. Reached out to Databricks support for help on this, will reply back if they help me find a resolution to the problem. Using the deprecated values for now.

Re: Dynamic Value References Not Working

themattmorris — Tue, 14 Nov 2023 15:58:27 GMT

Thanks @Retired_mod for the reply on reasons this may not be working. Regarding these potential problems, there is nothing in the documentation that suggests a minimum required version. But if you're aware of a minimum version requirement, do you mind sharing what that is? It would be helpful to have this in the documentation as well, if such a requirement exists. I've tried this on clusters with runtime version as late as 14.2 beta and have encountered the same problem on these as well.

Regarding a potential syntax error, the syntax in the original post follows what is provided in the documentation. But if you're aware of a potential syntax error in that code, do you mind commenting on that?

Thank you!

Re: Dynamic Value References Not Working

aseufert — Wed, 15 Nov 2023 13:21:26 GMT

Thanks for the reply. I'm using version webapp_2023-11-11_20.37.23Z_webappp_hotfix_20231106_ae45c0ec_1291847197

The jobs UI version is jobs-ui_2023-11-01_03.12.26Z_master_a08a2c30_1606137368

As @themattmorris said, I don't see the version mentioned anywhere in the documentation.

I also don't see the conditional task runs feature mentioned in the documentation so I assume I'm just on an old version of Databricks. Are you able to clarify which version of Databricks is required for this feature?

Thanks!

Re: Dynamic Value References Not Working

themattmorris — Thu, 16 Nov 2023 14:18:58 GMT

As I mentioned in an earlier post, I had been in contact with someone from Databricks regarding this problem. They got back to me yesterday indicating that Databricks actually still had to roll the feature out yesterday. This was done, and it is working now for me. I'd check again and see if these new reference values are working for you as well.

Re: Dynamic Value References Not Working

aseufert — Thu, 16 Nov 2023 14:46:01 GMT

@themattmorris Thanks for the update! I did see an alert that a new version of the job page was available. I was hoping it was the new features and sure enough, it was. I can confirm they're working for me as well.

Have a great day!

Re: Dynamic Value References Not Working

themattmorris — Thu, 16 Nov 2023 17:36:14 GMT

For what it's worth, it looks like job-level parameters were added with this update as well. I was wondering why I was unable to use those, but those are also working for me now.