Options
- Mark as New
- Bookmark
- Subscribe
- Mute
- Subscribe to RSS Feed
- Permalink
- Report Inappropriate Content
02-16-2024 01:42 AM
Hello Kaniz,
Thank you for your comprehensive answer.
Unfortunately none of those points apply to my case. The selected column is present exactly once in the source table and there is no more code, this is all I am running to reproduce the issue:
data = ( spark.read.format("delta")
.option("readChangeFeed", "true")
.option("startingVersion", 161)
.option("endingVersion", 161)
.table("table_name")
.select("GJAHR")
)
data.count()I just switch between 2 computes, one Single User and one Shared, both running on the same DBR 14.3, and I get the error only with the Shared cluster.
Thank you.