Options
- Mark as New
- Bookmark
- Subscribe
- Mute
- Subscribe to RSS Feed
- Permalink
- Report Inappropriate Content
05-23-2024 09:49 PM
@ha2983 I can confirm, that I can recreate the issue with your notebook.
In my case the error [NOT_COLUMN] Argument `col` should be a Column, got Column. occurs, when calling
.withColumn("IngestionTimestamp", unix_timestamp()) on a dataframe.
I can reproduce this error using the example from https://spark.apache.org/docs/3.5.0/api/python/reference/pyspark.sql/api/pyspark.sql.functions.unix_... and a single user cluster (DBR 14.3 LTS):
from pyspark.sql.functions import unix_timestamp
spark.conf.set("spark.sql.session.timeZone", "America/Los_Angeles")
spark.conf.set("spark.sql.session.timeZone", "America/Los_Angeles")
time_df = spark.createDataFrame([('2015-04-08',)], ['dt'])
time_df.select(unix_timestamp('dt', 'yyyy-MM-dd').alias('unix_time')).collect() spark.conf.unset("spark.sql.session.timeZone")
>>> [NOT_COLUMN_OR_STR] Argument `col` should be a Column or str, got Column.
On a shared cluster the code above works.
@dbruehlmeier we are also using vscode for local development and create our spark session like this:
from databricks.connect import DatabricksSession
spark = DatabricksSession.builder.getOrCreate()
