What exactly is Vectorized query processing and columnar acceleration

[dvl_priyansh]

Hey folks! I want to know and understand while using photon acceleration, there is a feature called columnar acceleration which basically is a method of storing data in columns rather than rows, which is particularly advantageous for analytical databases and data warehouses. I want to know that how it actually works and what makes it efficient during the ETL process?

Where as Vectorized query processing operates on batches of rows (vectors) rather than processing each row individually but how it actually makes a difference, whether we read the data in batches of by rows individually, anyhow we are reading each row only, so what makes there two features efficient in the case of photon acceleration?

[szymon_dybczak]

Hi @dvl_priyansh ,

Take a look at below article. It has a great explanation and answers you questions:

A Closer Look Into Databricks’s Photon Engine | by Vu Trinh | Data Engineer Things (det.life)

[Retired_mod]

Hi @szymon_dybczak, Thanks for reaching out! Please review the response and let us know if it answers your question. Your feedback is valuable to us and the community.

If the response resolves your issue, kindly mark it as the accepted solution. This will help close the thread and assist others with similar queries.

We appreciate your participation and are here if you need further assistance!

[szymon_dybczak]

Hi @Retired_mod ,

By mistake you've mentioned my name, but it was @dvl_priyansh  who actually asked a question 🙂

I've only provided an answer 😉