cancel
Turn on suggestions
Auto-suggest helps you quickly narrow down your search results by suggesting possible matches as you type.
Showing results for 
Search instead for 
Did you mean: 
Help
Data Engineering
Join discussions on data engineering best practices, architectures, and optimization strategies within the Databricks Community. Exchange insights and solutions with fellow data engineers.
cancel
Turn on suggestions
Auto-suggest helps you quickly narrow down your search results by suggesting possible matches as you type.
Showing results for 
Search instead for 
Did you mean: 

Forum Posts

Start a conversation
Abhot
by • New Contributor II
  • 5810 Views
  • 4 replies
  • 0 kudos

Temp Table Vs Temp View Vs temp table function- which one is better for large Databrick data processing

Hello , 1 ) Which one is better during large data processing - Temp table vs Temporary view vs temp Table function . 2) How lazy evaluation better for processing ? and which one of the above helps in lazy evaluation

Data Engineering
Reply
  • 5810 Views
  • 4 replies
  • 0 kudos
View Replies
Latest Reply
Abhot
New Contributor II
  • 0 kudos

Does anyone have any suggestions regarding the question above?

  • 0 kudos
3 More Replies
User15787040559
by • Databricks Employee
  • 27048 Views
  • 2 replies
  • 8 kudos

What's the difference between a Global view and a Temp view?

The difference between Global and Temp is how the lifetime of the view is tied to the application:http://spark.apache.org/docs/latest/api/python/reference/api/pyspark.sql.DataFrame.createOrReplaceTempView.html?highlight=createorreplacetempview#pyspar...

Data Engineering
Reply
  • 27048 Views
  • 2 replies
  • 8 kudos
View Replies
Latest Reply
ScottSmithDB
Databricks Employee
  • 8 kudos

Correct A Temp View is scoped to the SparkSession and dropped when that session closes.  Each notebook runs in its own SparkSession.  The Global Temp View is scoped to the cluster and dropped when the cluster re-starts or you drop it. ---------------...

  • 8 kudos
1 More Replies
HappySK
by • New Contributor
  • 5449 Views
  • 1 replies
  • 0 kudos

Execute COPY INTO query with source as temp view

We are having the files that needs to be loaded into the delta tableNow we want to perform some transformation on the files and load that into the tableWhat we didCreate a Spark DF from that fileApply transformation on the DFCreate a temp view from t...

Data Engineering
Reply
  • 5449 Views
  • 1 replies
  • 0 kudos
View Replies
Latest Reply
Ajay-Pandey
Esteemed Contributor III
  • 0 kudos

Hi @Sravan Kumar Mohanraj​ ,Yes, you can use copy query in this case your temp_view will be source.For more info, please visit these links.

  • 0 kudos
User16790091296
by • Contributor II
  • 15305 Views
  • 6 replies
  • 1 kudos

How to delete from a temp view or equivalent in spark sql databricks?

I need to delete from a temp view in databricks, but it looks like i can do only merge, select and insert. Maybe i missed something but I did not find any documentation on this.

Data Engineering
Reply
  • 15305 Views
  • 6 replies
  • 1 kudos
View Replies
Latest Reply
crazy_horse
New Contributor II
  • 1 kudos

What about%sqlDROP TABLE IF EXISTS xxxxx

  • 1 kudos
5 More Replies
manasa
by • Contributor
  • 4455 Views
  • 7 replies
  • 2 kudos

Resolved! Recursive view error while using spark 3.2.0 version

This happens while creating temp view using below code blocklatest_data.createOrReplaceGlobalTempView("e_test")ideally this command should replace the view if e_test already exists instead it is throwing"Recursive view `global_temp`.`e_test` detecte...

Data Engineering
Reply
  • 4455 Views
  • 7 replies
  • 2 kudos
View Replies
Latest Reply
shan_chandra
Databricks Employee
  • 2 kudos

Hi, @Manasa​, could you please check SPARK-38318 and use Spark 3.1.2, Spark 3.2.2, or Spark 3.3.0 to allow cyclic reference?

  • 2 kudos
6 More Replies
Labels