This code works in Scala only, there is no implementation for it in the Python runtime. And the information I got from a Databricks employee via email, those are internal APIs and they may change without notice.