topic Re: How Photon Acceleration Actually works? in Data Engineering

How Photon Acceleration Actually works?

priyansh — Wed, 25 Sep 2024 06:59:28 GMT

Hey folks!
I would like to know that how photon acceleration actually works, I have tested it on a sample of 219MB, 513MB, 2.7 GB, 4.1 GB of Data and the difference in seconds between normal and photon accelerated compute was not so much, So my question is, is it really worth to enable photon for reading and writing the data and if it is so for what data size and from what data size?
One more thing I would like to ask is the data is in parquet or delta format will there be a time reduction in photon compute comparatively to a normal compute
Will be looking forward to the answers
Regards,
Priyansh

Re: How Photon Acceleration Actually works?

arch_db — Wed, 25 Sep 2024 08:50:56 GMT

Try to check merge operation on tables over 200GB.

Re: How Photon Acceleration Actually works?

priyansh — Wed, 25 Sep 2024 08:56:43 GMT

Hey @arch_db
I will try it, can you please also tell me how Vectorized query processing helps in this, if the tales are either being read in batch or by row by row, in batch also each row should be read, so how it can help in time efficiency?

Re: How Photon Acceleration Actually works?

arch_db — Wed, 25 Sep 2024 09:08:01 GMT

Photon enable dynamic file pruning.

About DFP speedup you can watch in https://www.youtube.com/watch?v=yZmrpXJg-G8