Databricks

StevenW · 02-09-2023

I need to read/query table A, manipulate/modify the data and insert the new data into Table A again.I considered using :Cur_Actual = spark.sql("Select * from Table A")currAct_Rows = Cur_Actual.rdd.collect()for row in currAct_Rows: do_somthing(row)...

StevenW · 01-26-2023

I'm new to databricks. (Not new to DB's - 10+ year DB Developer).How do you generate a MERGE statement in DataBricks? Trying to manually maintain a 500+ or 1000+ lines in a MERGE statement doesn't make much sense? Working with Large Tables of between...

StevenW · 02-09-2023

Table A is a Delta table. I get this:Cur_Actual.write.format('delta').mode('append').save('/location/Table A')But as I understand it, one cannot loop over a DF, and hence the data is changed with the .collect() function to a collection.This data need...

StevenW · 01-27-2023

Thanks, this does make sense.I have a new lead to chase .. Much appreciated. 

StevenW · 01-27-2023

Would UPDATE SET * not require the the source and target columns to have the same names, and the same column order ?

StevenW · 01-27-2023

Yes, I mean the list of columns becomes large. To then maintain a MERGE statement could be very cumbersome.Would you happen to have an example of such Python code? This does actually make sense.. if such a list was generated dynamically, and then use...

StevenW · 01-27-2023

Interesting...I had not yet considered the performance issues. I didn't think there would be any .. will clearly need to come back to this one... I'm currently only concerned with the maintenance of a piece of code (the 500+ lines of the MERGE stat...

Databricks

User Stats

User Activity

Manipulating Data - using Notebooks

Large MERGE Statements - 500+ lines of code!

Re: Manipulating Data - using Notebooks

Re: Large MERGE Statements - 500+ lines of code!

Re: Large MERGE Statements - 500+ lines of code!

Re: Large MERGE Statements - 500+ lines of code!

Re: Large MERGE Statements - 500+ lines of code!