cancel
Showing results for 
Search instead for 
Did you mean: 
Data Engineering
cancel
Showing results for 
Search instead for 
Did you mean: 

Forum Posts

sage5616
by Valued Contributor
  • 3101 Views
  • 3 replies
  • 4 kudos

Resolved! Spark persistent view on a partition parquet file

In Spark, is it possible to create a persistent view on a partitioned parquet file in Azure BLOB? The view must be available when the cluster restarted, without having to re-create that view, hence it cannot be a temp view.I can create a temp view, b...

  • 3101 Views
  • 3 replies
  • 4 kudos
Latest Reply
sage5616
Valued Contributor
  • 4 kudos

Here is what worked for me. Hope this helps someone else: https://stackoverflow.com/questions/72913913/spark-persistent-view-on-a-partition-parquet-file/72914245#72914245CREATE VIEW test as select * from parquet.`/mnt/folder-with-parquet-file(s)/`@Hu...

  • 4 kudos
2 More Replies
bdugar
by New Contributor II
  • 9202 Views
  • 2 replies
  • 2 kudos

Creating permanent views from dataframes?

Hi:It's possible to create temp views in pyspark using a dataframe (df.createOrReplaceTempView()), and it's possible to create a permanent view in Spark SQL. But as far as I can tell, there is no way to create a permanent view from a dataframe, somet...

  • 9202 Views
  • 2 replies
  • 2 kudos
Latest Reply
bdugar
New Contributor II
  • 2 kudos

Hi Kaniz:This is what I understood from the research I did, I was curious more as to why permanent views can't be created from dataframes and whether this is a feature that might be implemented by Databricks or Spark at some point. Temporary views ca...

  • 2 kudos
1 More Replies
Labels