cancel
Showing results for 
Search instead for 
Did you mean: 
Data Engineering
Join discussions on data engineering best practices, architectures, and optimization strategies within the Databricks Community. Exchange insights and solutions with fellow data engineers.
cancel
Showing results for 
Search instead for 
Did you mean: 

How Photon Acceleration Actually works?

priyansh
New Contributor III

Hey folks!
I would like to know that how photon acceleration actually works, I have tested it on a sample of 219MB, 513MB, 2.7 GB, 4.1 GB of Data and the difference in seconds between normal and photon accelerated compute was not so much, So my question is, is it really worth to enable photon for reading and writing the data and if it is so for what data size and from what data size?
One more thing I would like to ask is the data is in parquet or delta format will there be a time reduction in photon compute comparatively to a normal compute 
Will be looking forward to the answersimage (4).png
Regards,
Priyansh

3 REPLIES 3

arch_db
New Contributor II

Try to check merge operation on tables over 200GB.

priyansh
New Contributor III

Hey @arch_db 
I will try it, can you please also tell me how Vectorized query processing helps in this, if the tales are either being read in batch or by row by row, in batch also each row should be read, so how it can help in time efficiency? 

arch_db
New Contributor II

Photon enable dynamic file pruning.

About DFP speedup you can watch in https://www.youtube.com/watch?v=yZmrpXJg-G8

This session explores optimizing MERGE performance using Liquid Clustering and Photon. It explores how you can use recent Delta Lake and Photon features such as Liquid Clustering, Deletion Vectors, and Predictive I/O for Updates to make your MERGEs run smoothly and with predictable performance ...

Connect with Databricks Users in Your Area

Join a Regional User Group to connect with local Databricks users. Events will be happening in your city, and you won’t want to miss the chance to attend and share knowledge.

If there isn’t a group near you, start one and help create a community that brings people together.

Request a New Group