Options
- Mark as New
- Bookmark
- Subscribe
- Mute
- Subscribe to RSS Feed
- Permalink
- Report Inappropriate Content
05-23-2024 07:55 AM
This Notebook can be used to recreate the issue:
import pandas as pd
from databricks.connect import DatabricksSession
from pyspark.sql.functions import current_timestamp
spark = DatabricksSession.builder.getOrCreate()
# Create a pandas DataFrame
data = {
"Name": ["John", "Alice", "Bob"],
"Age": [25, 30, 35],
"City": ["New York", "San Francisco", "Los Angeles"],
}
df = pd.DataFrame(data)
# Convert pandas DataFrame to Spark DataFrame
spark_df = spark.createDataFrame(df)
spark_df = spark_df.withColumn("_loaded_at", current_timestamp())
spark_df.show()I used databricks runtime 14.3 LTS with single user access mode