Lakehouse Federation Join Pushdown

[Schaubi]

Hi,

I experimented a little bit with lakehouse federation. I created a connection and foreign catalog that references a SQL Server and activated the public preview feature for Join Pushdowns. After finishing my experiments, it seems to me that the feature has the limitation that cross-schema joins are not pushed down to SQL Server. I could not find any documentation about this limitation (https://learn.microsoft.com/en-us/azure/databricks/query-federation/performance-recommendations#join...), so I wanted to ask if someone knows, if this is a known limitation or if my assumption might be false and the problem lies elsewhere.

Example where the Join is pushed down:

select *
from catalog_name.schema1.table_x tx
inner join catalog_name.schema1.table_y ty
    on tx.join_key = ty.join_key

Example where the join is NOT pushed down:

select *
from catalog_name.schema1.table_x tx
inner join catalog_name.schema2.table_y ty
    on tx.join_key = ty.join_key

I'm using Databricks Runtime 17.3 for my experiments.

Thanks,

Stefan

[MoJaMa · Accepted solution]

I checked with Engineering. Unfortunately this is not supported and they have an action item to update the docs to clarify it for others.