Re: Cannot use join with Enzyme

sebih · ‎08-06-2025

I suppose I can use incrementalization on pipelines. Supported operators are listed in here: https://docs.databricks.com/aws/en/optimizations/incremental-refresh#support-for-materialized-view-i...However, when I run the pipeline, it does not use incrementalization saying that "the inner join operator" is not supported as opposed to the documentation.

{
  "maintenance_type": "MAINTENANCE_TYPE_WINDOW_FUNCTION",
  "incrementalization_issues": [
    {
      "issue_type": "PLAN_NOT_INCREMENTALIZABLE",
      "prevent_incrementalization": true,
      "operator_name": "Join",
      "join_type": "INNER",
      "plan_not_incrementalizable_sub_type": "OPERATOR_NOT_SUPPORTED"
    }
  ]
},

I've also set all base tables' properties to this:

delta.enableDeletionVectors = true,
delta.enableRowTracking = true,
delta.enableChangeDataFeed = true

I wonder whether there is a switch on your side to enable this. Is it possible? I just want to make clear the conflict that is created by the product and its documentation.

Louis_Frolio · ‎08-06-2025

I've seen anecdotally that if you join more than two tables, that particular error will be thrown even though it technically supports joins. I suggest reaching out to Databricks support, there may be something they can do with hidden flags such as "pipelines.enzyme.numberOfJoinsThreshold."

Cheers, Louis.

sebih · ‎08-14-2025

Thank you for your reply. Even though we only do one join, we keep getting this error.