Options
- Mark as New
- Bookmark
- Subscribe
- Mute
- Subscribe to RSS Feed
- Permalink
- Report Inappropriate Content
11-30-2022 03:27 AM
Hi @Nabil AMEZIANE ,
What you were doing works actually...
You just need to fetch the latest version into a variable like below
x= spark.sql("""select max(version) from (desc HISTORY <table-name>)""").collect()[0][0]
Be sure to check the operation type too in the history table if there are DELETE operations.
Now you have a integer x with latest version number available. For previous version all you need to do is to do a x-1. Then you have both version numbers.
Now pass these variables into your sql query reading those versions. I would do it as below
spark.sql("""
select * from table @v{0}
minus
select * from table @v{1}
""".format(x, x-1))Hope this helps...Cheers
Uma Mahesh D