This is another great tip. Thank you for sharing!

One small follow-up, I looked into this a while back but didn't have luck with it at the time. Regarding variables in Spark SQL, is there a way to initialize a variable outside of the %sql cell? For example, let's say I initialize a variable in Python, but I want to use the variable in %sql..

Is this possible?