I'm using the following input parameters when running from a workflow:wid_UnityCatalogName = dbutils.jobs.taskValues.get(taskKey="NB_XXX_Workflow_Parameters", key="p_UnityCatalogName", default="xx_lakehouse_dev") dbutils.widgets.text("UnityCat...
I need to read/query table A, manipulate/modify the data and insert the new data into Table A again.I considered using :Cur_Actual = spark.sql("Select * from Table A")currAct_Rows = Cur_Actual.rdd.collect()for row in currAct_Rows: do_somthing(row)...
I'm new to databricks. (Not new to DB's - 10+ year DB Developer).How do you generate a MERGE statement in DataBricks? Trying to manually maintain a 500+ or 1000+ lines in a MERGE statement doesn't make much sense? Working with Large Tables of between...
Hi @filipnizioThanks .. yes, I'm using the %sql command. I used ${xx_lakehouse_dev} and $xx_lakehouse_dev .. for most of the sql in the same notebook, it works fine. Then "suddenly" it doesn't ..?
Table A is a Delta table. I get this:Cur_Actual.write.format('delta').mode('append').save('/location/Table A')But as I understand it, one cannot loop over a DF, and hence the data is changed with the .collect() function to a collection.This data need...
Yes, I mean the list of columns becomes large. To then maintain a MERGE statement could be very cumbersome.Would you happen to have an example of such Python code? This does actually make sense.. if such a list was generated dynamically, and then use...
