Observable API and Delta Table merge

Malthe
Valued Contributor II

Using the Observable API on the source dataframe to a Delta Table merge seems to hang indefinitely.

Steps to reproduce:

  1. Create one or more pyspark.sql.Observation objects.
  2. Use DataFrame.observe on the merge source.
  3. Run merge.
  4. Accessing Observation.get blocks indefinitely.

The source dataframe here is a batch dataframe, executed within the foreachBatch framework on a streaming data source.

Is the Observable API not compatible with Delta Table merges?