https://community.databricks.com/t5/data-engineering/photon-does-not-fully-support-the-query-because-of-dynamic/m-p/22543#M15449 <P>Does it still make sense to run this job on a cluster with Photon enable when I am receiving the following?</P><P></P><P>This is the code I ran:</P><PRE><CODE>CREATE OR REPLACE TABLE ${tbl_name}_dups SELECT src.*, ROW_NUMBER() OVER ( PARTITION BY src.id ORDER BY src.id ) row_num FROM ${tbl_name} src INNER JOIN (select id, count(*) n from ${tbl_name} group by id having n &gt; 1) dups ON src.id = dups.id;</CODE></PRE><P>This is what I see under the cluster UI: SQL/Dataframe section</P><PRE><CODE>== Photon Explanation == Photon does not fully support the query because: Unsupported expression(s): dynamicpruning#601 600 reference node: FileScan parquet spark_catalog.default.opensea_events[approved_account#466,asset#467,asset_bundle#468,auction_type#469,bid_amount#470,collection_slug#471,contract_address#472,created_date#473,custom_event_name#474,dev_fee_payment_event#475,dev_seller_fee_basis_points#476L,duration#477,ending_price#478,event_timestamp#479,from_account#481,id#482L,is_private#483,listing_time#484,owner_account#485,payment_token#486,quantity#487,seller#488,starting_price#489,to_account#490,... 5 more fields] Batched: true, DataFilters: [isnotnull(id#482L), dynamicpruning#601 600], Format: Parquet, Location: PreparedDeltaFileIndex(1 paths)[dbfs:/mnt/opensea-tbl/asset_events_2], PartitionFilters: [], PushedFilters: [IsNotNull(id)], ReadSchema: struct&lt;approved_account:string,asset:struct&lt;animation_original_url:string,animation_url:string,as...</CODE></PRE><P></P> Sun, 13 Nov 2022 22:59:53 GMT lawrence009 2022-11-13T22:59:53Z https://community.databricks.com/t5/data-engineering/photon-does-not-fully-support-the-query-because-of-dynamic/m-p/22543#M15449 <P>Does it still make sense to run this job on a cluster with Photon enable when I am receiving the following?</P><P></P><P>This is the code I ran:</P><PRE><CODE>CREATE OR REPLACE TABLE ${tbl_name}_dups SELECT src.*, ROW_NUMBER() OVER ( PARTITION BY src.id ORDER BY src.id ) row_num FROM ${tbl_name} src INNER JOIN (select id, count(*) n from ${tbl_name} group by id having n &gt; 1) dups ON src.id = dups.id;</CODE></PRE><P>This is what I see under the cluster UI: SQL/Dataframe section</P><PRE><CODE>== Photon Explanation == Photon does not fully support the query because: Unsupported expression(s): dynamicpruning#601 600 reference node: FileScan parquet spark_catalog.default.opensea_events[approved_account#466,asset#467,asset_bundle#468,auction_type#469,bid_amount#470,collection_slug#471,contract_address#472,created_date#473,custom_event_name#474,dev_fee_payment_event#475,dev_seller_fee_basis_points#476L,duration#477,ending_price#478,event_timestamp#479,from_account#481,id#482L,is_private#483,listing_time#484,owner_account#485,payment_token#486,quantity#487,seller#488,starting_price#489,to_account#490,... 5 more fields] Batched: true, DataFilters: [isnotnull(id#482L), dynamicpruning#601 600], Format: Parquet, Location: PreparedDeltaFileIndex(1 paths)[dbfs:/mnt/opensea-tbl/asset_events_2], PartitionFilters: [], PushedFilters: [IsNotNull(id)], ReadSchema: struct&lt;approved_account:string,asset:struct&lt;animation_original_url:string,animation_url:string,as...</CODE></PRE><P></P> Sun, 13 Nov 2022 22:59:53 GMT https://community.databricks.com/t5/data-engineering/photon-does-not-fully-support-the-query-because-of-dynamic/m-p/22543#M15449 lawrence009 2022-11-13T22:59:53Z https://community.databricks.com/t5/data-engineering/photon-does-not-fully-support-the-query-because-of-dynamic/m-p/22544#M15450 <P>hmm could you show us what your query is</P> Mon, 14 Nov 2022 00:04:17 GMT https://community.databricks.com/t5/data-engineering/photon-does-not-fully-support-the-query-because-of-dynamic/m-p/22544#M15450 PriyaAnanthram 2022-11-14T00:04:17Z https://community.databricks.com/t5/data-engineering/photon-does-not-fully-support-the-query-because-of-dynamic/m-p/22545#M15451 <P>Hi @Priya Ananthram​, I amended my original post.</P> Mon, 14 Nov 2022 23:48:23 GMT https://community.databricks.com/t5/data-engineering/photon-does-not-fully-support-the-query-because-of-dynamic/m-p/22545#M15451 lawrence009 2022-11-14T23:48:23Z https://community.databricks.com/t5/data-engineering/photon-does-not-fully-support-the-query-because-of-dynamic/m-p/22547#M15453 <P>Hi @Kaniz Fatma​,&nbsp;DBR 11.3</P> Tue, 22 Nov 2022 02:44:03 GMT https://community.databricks.com/t5/data-engineering/photon-does-not-fully-support-the-query-because-of-dynamic/m-p/22547#M15453 lawrence009 2022-11-22T02:44:03Z