Photon is not supported for a query

[sukanya09]

(1) LocalTableScan
Output [11]: [path#23524, partitionValues#23525, size#23526L, modificationTime#23527L, dataChange#23528, stats#23529, tags#23530, deletionVector#23531, baseRowId#23532L, defaultRowCommitVersion#23533L, clusteringProvider#23534]
Arguments: [path#23524, partitionValues#23525, size#23526L, modificationTime#23527L, dataChange#23528, stats#23529, tags#23530, deletionVector#23531, baseRowId#23532L, defaultRowCommitVersion#23533L, clusteringProvider#23534]


== Photon Explanation ==
Photon does not fully support the query because:
		Unsupported node: LocalTableScan [path#23524, partitionValues#23525, size#23526L, modificationTime#23527L, dataChange#23528, stats#23529, tags#23530, deletionVector#23531, baseRowId#23532L, defaultRowCommitVersion#23533L, clusteringProvider#23534].

Reference node:
	LocalTableScan [path#23524, partitionValues#23525, size#23526L, modificationTime#23527L, dataChange#23528, stats#23529, tags#23530, deletionVector#23531, baseRowId#23532L, defaultRowCommitVersion#23533L, clusteringProvider#23534]

Any idea how we can improve here ? 

[Kaniz_Fatma]

Hi @sukanya09, 

• The query you provided includes a LocalTableScan node, which Photon does not fully support.
• The specific node you mentioned has several attributes, such as path, partitionValues, size, modificationTime, and more.
• Unfortunately, Photon encounters limitations when dealing with this type of node.
• The reason for the lack of full support is not specified, but it likely relates to Photon’s limitations or compatibility constraints.
• To improve the query, consider the following steps:
  • Optimize the Query Logic:
    • Review your query and see if there are any unnecessary LocalTableScan operations.
    • If possible, replace them with more efficient operations.
  • Check for Dynamic Pruning:
    • Sometimes, unsupported expressions like dynamic pruning can cause issues.
    • Ensure that your query doesn’t rely on unsupported features.
  • Analyze Query Execution:
    • Use the Databricks UI to analyze the execution details of your query.
    • Look at the “Task Time in Photon” metric to identify bottlenecks.
    • Consider rewriting parts of your query to avoid problematic nodes.
• If you have any specific queries or need further assistance, feel free to ask! 😊