Databricks Community

dbal · 05-10-2024

I am not able to run our unit tests suite due a possible bug in the databricks-connect library. The problem is with the Dataframe transformation withColumnRenamed. When I run it in a Databricks cluster (Databricks Runtime 14.3 LTS), the column is ren...

dbal · 03-13-2024

Hi.I am trying to run a Spark Job in Databricks (Azure) using the JAR type.I can't figure out why the job fails to run by not finding the SparkContext.Databricks Runtime: 14.3 LTS (includes Apache Spark 3.5.0, Scala 2.12)Error message: java.lang.NoCl...

dbal

Any update on this? I am also facing this issue.

dbal · 05-17-2024

Hi @shan_chandra thank you for your answer.I am using Scala, not Python. Nevertheless, I tried using the withColumnsRenamed(), but sadly it shows the same issue.

dbal · 03-18-2024

Thanks for providing a solution @pokus .What I dont understand is why Databricks cannot provide the DeltaLog at runtime. How can this be the official solution? We need a better solution for this instead of depending on reflections.

dbal · 03-13-2024

Update 2: I found the reason in the documentation. This is documented under "Access Mode", and it is a limitation of the Shared access mode.Link: https://learn.microsoft.com/en-us/azure/databricks/compute/access-mode-limitations#spark-api-limitations...

dbal · 03-13-2024

Update 1: It works when I change the Runtime to 14.3 LTS ML (includes Apache Spark 3.5.0, Scala 2.12)

Databricks Community

User Stats

User Activity

withColumnRenamed does not work with databricks-connect 14.3.0

Spark job task fails with "java.lang.NoClassDefFoundError: org/apache/spark/SparkContext$"

Re: Show log4j messages in run output

Re: withColumnRenamed does not work with databricks-connect 14.3.0

Re: use DeltaLog class in databricks cluster

Re: Spark job task fails with "java.lang.NoClassDefFoundError: org/apache/spark/SparkContext$&q

Re: Spark job task fails with "java.lang.NoClassDefFoundError: org/apache/spark/SparkContext$&q