Observable API and Delta Table merge
Options
- Mark as New
- Bookmark
- Subscribe
- Mute
- Subscribe to RSS Feed
- Permalink
- Report Inappropriate Content
03-11-2026 04:01 AM
Using the Observable API on the source dataframe to a Delta Table merge seems to hang indefinitely.
Steps to reproduce:
- Create one or more pyspark.sql.Observation objects.
- Use DataFrame.observe on the merge source.
- Run merge.
- Accessing Observation.get blocks indefinitely.
The source dataframe here is a batch dataframe, executed within the foreachBatch framework on a streaming data source.
Is the Observable API not compatible with Delta Table merges?