Databricks Community

ahsan_aj

Hi All,I am using Databricks Connect 14.3.2 with Databricks Runtime 14.3 LTS to execute the code below. The CSV file is only 7MB, the code runs without issues on Databricks Runtime 15+ clusters but consistently produces the error message shown below ...

ahsan_aj

Databricks confirmed the same workaround while they work on a permanent fix.

ahsan_aj

As a workaround, please try the following Spark configuration, which seems to have resolved the issue for me on both 14.3 LTS and 15.4 LTS.spark.conf.set("spark.sql.session.localRelationCacheThreshold", 64 * 1024 * 1024)

ahsan_aj

I have a troubleshooting session scheduled with Databricks today regarding this issue and will keep everyone updated on the progress.

ahsan_aj

Microsoft support just mentioned that fix has been deployed by Databricks, but the issue continues to persist for me on both 14.3 LTS and 15.4 LTS.

ahsan_aj

It looks like there is no update on this, this is such a useful piece of information that ideally should have been available from the start.

Databricks Community

User Stats

User Activity

Databricks connect 14.3.2 SparkConnectGrpcException Not found any cached local relation withthe hash

Re: Databricks connect 14.3.2 SparkConnectGrpcException Not found any cached local relation withthe

Re: Databricks connect 14.3.2 SparkConnectGrpcException Not found any cached local relation withthe

Re: Databricks connect 14.3.2 SparkConnectGrpcException Not found any cached local relation withthe

Re: Databricks connect 14.3.2 SparkConnectGrpcException Not found any cached local relation withthe

Re: How to know, who started a job run?