- 5207 Views
- 3 replies
- 4 kudos
Resolved! Spark persistent view on a partition parquet file
In Spark, is it possible to create a persistent view on a partitioned parquet file in Azure BLOB? The view must be available when the cluster restarted, without having to re-create that view, hence it cannot be a temp view.I can create a temp view, b...
- 5207 Views
- 3 replies
- 4 kudos
- 4 kudos
Here is what worked for me. Hope this helps someone else: https://stackoverflow.com/questions/72913913/spark-persistent-view-on-a-partition-parquet-file/72914245#72914245CREATE VIEW test as select * from parquet.`/mnt/folder-with-parquet-file(s)/`@Hu...
- 4 kudos
- 16507 Views
- 1 replies
- 2 kudos
Creating permanent views from dataframes?
Hi:It's possible to create temp views in pyspark using a dataframe (df.createOrReplaceTempView()), and it's possible to create a permanent view in Spark SQL. But as far as I can tell, there is no way to create a permanent view from a dataframe, somet...
- 16507 Views
- 1 replies
- 2 kudos
- 2 kudos
Hi Kaniz:This is what I understood from the research I did, I was curious more as to why permanent views can't be created from dataframes and whether this is a feature that might be implemented by Databricks or Spark at some point. Temporary views ca...
- 2 kudos