smukhi
New Contributor II

As advised, I double confirmed that no code or cluster configuration was changed (even got a second set of eyes on it that confirmed the same).

I was able to find a "fix" which puts a bandaid on the issue:

I was able to pinpoint that the issue seems to only occur when I checkpoint the DataFrame using "df.checkpoint(eager=True)". After replacing this checkpoint with a write to S3 using "df.write.parquet(...)", followed by a read back from the same location "df = spark.read.parquet(...)", I'm able to circumvent the above error message while still creating a "checkpoint" without using the df.checkpoint method.

I'm still unsure how the Unity Catalog service pertains to the df.checkpoint method, or what may have changed last Friday with this functionality, as I'm unable to read the proprietary Databricks modules that appear in the stack trace. It may still be worth an investigation on your end!