Databricks

zak_k · 09-25-2023

Trying to determine a root cause of UDFException that occurs when returning a variable length ArrayType. If I hardcode the data returned from the UDF to a fixed length, say 19, the error does not occur. Setup codesplit_runs_UDF = udf(split_runs_udf, ...

zak_k · 09-27-2023

After further investigation, It reproduces slightly differently on single user mode.Single user mode: runs foreverShared: gives the above messageI've determined that there was a corner case in the dataset which lead to UDF never returning. I am am as...

zak_k · 09-26-2023

I have Confirmed that in small batches the UDF works on both single user and shared clusters, but with large data sets, it only works for single user clusters

zak_k · 09-26-2023

I'm being told it only occurs on shared mode clusters, I'm going to confirm this now.

zak_k · 09-26-2023

It's not currently part of DLT in this early phase, but that is the end goal. DBR version is 13.3.The use case is splitting some records into 2 or more records (when start and end time cross a shift work boundary).

Databricks

User Stats

User Activity

com.databricks.spark.safespark.UDFException: UNAVAILABLE: Channel shutdownNow invoked

Re: com.databricks.spark.safespark.UDFException: UNAVAILABLE: Channel shutdownNow invoked

Re: com.databricks.spark.safespark.UDFException: UNAVAILABLE: Channel shutdownNow invoked

Re: com.databricks.spark.safespark.UDFException: UNAVAILABLE: Channel shutdownNow invoked

Re: com.databricks.spark.safespark.UDFException: UNAVAILABLE: Channel shutdownNow invoked