topic How to unload a Jar for UDF without restart spark context? in Data Engineering

How to unload a Jar for UDF without restart spark context?

vonjack — Thu, 17 Aug 2023 09:38:38 GMT

In the scala notebook of databricks, I created a temporary function with a certain Jar and class name.

Then I want to update the Jar. But without restart the context, I can not reload the new Jar, the temporary function always reuses the old classes.

I noticed that for normal function it support refresh, but temporary function doesn't.

Does anyone know the way to resolve this issue?

Re: How to unload a Jar for UDF without restart spark context?

vonjack — Thu, 17 Aug 2023 10:28:20 GMT

Hi, I tried updated Jar, drop and re-create the temporary function, the UDF still use the old Jar.

I don't want to restart context and cleanup workspace, are there any other better solutions?

Re: How to unload a Jar for UDF without restart spark context?

vonjack — Fri, 18 Aug 2023 00:39:56 GMT

So if I don't want to restart the cluster, do you think the versionized jar name and class name would be a better solution?

At first, I used this command to create the original UDF:

create or replace temporary function test_udf as 'com.example.TestUDF_v1' using jar '/tmp/test_udf_v1.jar';

Then I want to update the UDF, I'll try that:

create or replace temporary function test_udf as 'com.example.TestUDF_v2' using jar '/tmp/test_udf_v2.jar';

But that may be into another problem, if different jars have the same class name like 'com.example.SameClass', which version of class will spark cluster use eventually without restart?