How to unload a Jar for UDF without restart spark context?

vonjack
New Contributor II

‎08-17-2023 02:38 AM

In the scala notebook of databricks, I created a temporary function with a certain Jar and class name. 

Then I want to update the Jar. But without restart the context, I can not reload the new Jar, the temporary function always reuses the old classes.

I noticed that for normal function it support refresh, but temporary function doesn't.

Does anyone know the way to resolve this issue?

Labels:
0 Kudos

vonjack
New Contributor II
In response to Meganabbott

‎08-17-2023 03:28 AM

Hi, I tried updated Jar, drop and re-create the temporary function, the UDF still use the old Jar.

I don't want to restart context and cleanup workspace, are there any other better solutions?

0 Kudos

vonjack
New Contributor II
In response to Retired_mod

‎08-17-2023 05:36 PM - edited ‎08-17-2023 05:39 PM

So if I don't want to restart the cluster, do you think the versionized jar name and class name would be a better solution?

 

At first, I used this command to create the original UDF:

create or replace temporary function test_udf as 'com.example.TestUDF_v1' using jar '/tmp/test_udf_v1.jar';

 

Then I want to update the UDF, I'll try that: 

create or replace temporary function test_udf as 'com.example.TestUDF_v2' using jar '/tmp/test_udf_v2.jar';

But that may be into another problem, if different jars have the same class name like 'com.example.SameClass', which version of class will spark cluster use eventually without restart?

 

0 Kudos