topic Re: DatabricksSession broken for 15.1 in Data Engineering

DatabricksSession broken for 15.1

TWib — Fri, 24 May 2024 10:40:00 GMT

This code fails with exception:

[NOT_COLUMN_OR_STR] Argument `col` should be a Column or str, got Column.

File <command-4420517954891674>, line 7  4 spark = DatabricksSession.builder.getOrCreate()  6 df = spark.read.table("samples.nyctaxi.trips") ----> 7 df.select(lit(5).alias('height')).show()

from databricks.connect import DatabricksSession from pyspark.sql.functions import lit spark = DatabricksSession.builder.getOrCreate() df = spark.range(1) df.select(lit(5).alias('height'), df.id).show()

Can you confirm this is a bug?

Re: DatabricksSession broken for 15.1

MM3 — Mon, 27 May 2024 14:20:03 GMT

It's an official example from pyspark documentation:

https://spark.apache.org/docs/latest/api/python/reference/pyspark.sql/api/pyspark.sql.functions.lit.html

It works on older runtime, it used to work one week ago. Please fix your internal databricks connect on latest runtimes.

Re: DatabricksSession broken for 15.1

TWib — Tue, 28 May 2024 05:33:38 GMT

I am sorry but this is not helpfull. ChatGPT does not work well for PySpark code

Re: DatabricksSession broken for 15.1

TWib — Tue, 28 May 2024 05:35:06 GMT

We dont understand the issue becauuse it suddently appeared but fixed it with migrating to 15.2.

Maybe databricks released some 15.1.XXXX update that broke stuff?

Re: DatabricksSession broken for 15.1

zerodarkzone — Wed, 29 May 2024 15:27:15 GMT

Hi, I'm having the same problem using the 14.3LTS runtime.

The error just appeared yesterday. Before that, everything was working fine.

Re: DatabricksSession broken for 15.1

jcap — Wed, 29 May 2024 21:48:23 GMT

We are also seeing this error in 14.3 LTS from a simple example:

from pyspark.sql.functions import col

df = spark.table('things')
things = df.select(col('thing_id')).collect()

[NOT_COLUMN_OR_STR] Argument `col` should be a Column or str, got Column.

Re: DatabricksSession broken for 15.1

977073 — Wed, 11 Sep 2024 00:51:09 GMT

I can see this issue in 13.3 LTS, production code still running in 11.3LTS but upgradding to higher LTS DBR version gives this error. I believe you should fix it or provide a migration guide from one DBR to the other

Re: DatabricksSession broken for 15.1

yigalk — Mon, 16 Dec 2024 16:13:51 GMT

I also get the same for runtime 13.3 LTS. The same code with 15.2 LTS seems to work.

df.withColumn("new_col", concat("col1", lit("-"), "col2"))

Re: DatabricksSession broken for 15.1

yigalk — Mon, 16 Dec 2024 18:47:59 GMT

So actually it isn't working on runtime 15.

It is working on a shared cluster in runtime 15.4. But then I need also to use rdd for something and it fails on shared clusters.