ha2983
New Contributor III

This Notebook can be used to recreate the issue:

import pandas as pd
from databricks.connect import DatabricksSession
from pyspark.sql.functions import current_timestamp

spark = DatabricksSession.builder.getOrCreate()


# Create a pandas DataFrame
data = {
    "Name": ["John", "Alice", "Bob"],
    "Age": [25, 30, 35],
    "City": ["New York", "San Francisco", "Los Angeles"],
}
df = pd.DataFrame(data)

# Convert pandas DataFrame to Spark DataFrame
spark_df = spark.createDataFrame(df)


spark_df = spark_df.withColumn("_loaded_at", current_timestamp())

spark_df.show()

I used databricks runtime 14.3 LTS with single user access mode