Creating permanent views from dataframes?

[bdugar]

Hi:

It's possible to create temp views in pyspark using a dataframe (df.createOrReplaceTempView()), and it's possible to create a permanent view in Spark SQL. But as far as I can tell, there is no way to create a permanent view from a dataframe, something like df.createView().

This is entirely confusing to me - clearly the environment supports permanent views, and it supports creating views from dataframes. There are a lot of scenarios where using python is more convenient for building complex queries instead of trying to construct a SQL string and execute it within a context.

Is support for this something that is eventually planned? Is Databricks beholden to whatever Spark decides to implement at this level? Is there a great reason I, someone who doesn't know much about the inner workings of Spark, am overlooking as to why this isn't possible or is very hard?

Thanks,

Bryce

[bdugar]

Hi Kaniz:

This is what I understood from the research I did, I was curious more as to why permanent views can't be created from dataframes and whether this is a feature that might be implemented by Databricks or Spark at some point. Temporary views can be created, and permanent views exist, so it seems like there's an odd gap between the two that has a lot of utility if filled. It'd be different if dataframes just couldn't create any view, or if permanent views didn't exist at all. But maybe there's something I don't know.

Also, if you intended to link something I don't see a clickable link or anything in your message, unsure what pull request you're referencing.