โ11-13-2021 09:40 AM
For context, I am running Spark on databricks platform and using Delta Tables (s3).
Let's assume we a table called table_one. I create a view called view_one using the table and then call view_one. Next, I create another view, called view_two based on view_one and then call view_two. Will all the calculations be done again for view_one.
Example commands are below i.e. when cmd4 is called, will cmd1 be re-executed to calculate cmd4?
Cmd1:
CREATE OR REPLACE VIEW_ONE FROM
SELECT
....
FROM
table_one
WHERE
.....
Cmd2:
SELECT * FROM VIEW_ONE;
Cmd3:
CREATE OR REPLACE VIEW VIEW_TWO AS
SELECT
....
FROM
VIEW_ONE
WHERE
.....;
Cmd4:
SELECT * FROM VIEW_TWO;
โ11-15-2021 11:08 AM
Hi @John Constantineโ ,
The following notebook url will help you to undertand better the difference between lazy transformations and action in Spark. You will be able to compare the physical query plans and undertand better what is going on when you execute your SQL statements.
โ11-13-2021 12:14 PM
Hello @John Constantineโ! My name is Piper and I'm a community moderator for Databricks. Welcome to the community and thank you for your question! Let's give it a while to see what other members have to say. ๐
โ11-13-2021 10:57 PM
short answer: yes. Spark will run view_one twice.
Unless you cache it (by using delta cache or persist()/cache()).
โ11-15-2021 09:33 AM
Hi @John Constantineโ for delta caching you can refer to the below doc link.
https://docs.databricks.com/delta/optimizations/delta-cache.html
โ11-15-2021 11:08 AM
Hi @John Constantineโ ,
The following notebook url will help you to undertand better the difference between lazy transformations and action in Spark. You will be able to compare the physical query plans and undertand better what is going on when you execute your SQL statements.
Join a Regional User Group to connect with local Databricks users. Events will be happening in your city, and you wonโt want to miss the chance to attend and share knowledge.
If there isnโt a group near you, start one and help create a community that brings people together.
Request a New Group