Creating permanent views from dataframes?

[bdugar]

Hi:

It's possible to create temp views in pyspark using a dataframe (df.createOrReplaceTempView()), and it's possible to create a permanent view in Spark SQL. But as far as I can tell, there is no way to create a permanent view from a dataframe, something like df.createView().

This is entirely confusing to me - clearly the environment supports permanent views, and it supports creating views from dataframes. There are a lot of scenarios where using python is more convenient for building complex queries instead of trying to construct a SQL string and execute it within a context.

Is support for this something that is eventually planned? Is Databricks beholden to whatever Spark decides to implement at this level? Is there a great reason I, someone who doesn't know much about the inner workings of Spark, am overlooking as to why this isn't possible or is very hard?

Thanks,

Bryce

[Kaniz]

Hi @Bryce Dugar​, Please check this pull request. It mentions that creating a permanent view that references a quick view is disallowed.

Yes, you are correct that PySpark has no methods to create a permanent view, e.g., createTempView and createOrReplaceTempView. You can create only a temporary view. But you can run SQL queries on top of that.

[bdugar]

Hi Kaniz:

This is what I understood from the research I did, I was curious more as to why permanent views can't be created from dataframes and whether this is a feature that might be implemented by Databricks or Spark at some point. Temporary views can be created, and permanent views exist, so it seems like there's an odd gap between the two that has a lot of utility if filled. It'd be different if dataframes just couldn't create any view, or if permanent views didn't exist at all. But maybe there's something I don't know.

Also, if you intended to link something I don't see a clickable link or anything in your message, unsure what pull request you're referencing.