Ok, so for testing purposes remove state_path mapping from yaml path. The default path for state shoud be ${workspace.root}/state.Also, you can run enable debug mode in databricks cli. Maybe we can find some useful information from execution:Trouble...
As @Walter_C mentioned, merge is a a proper way to perform upsert. I just want to add that in Databricks primary and foreign constraints are informational only constraints and thus are not enforced. So, @ChristianRRL be aware of that fact, because I...
Hi @genevive_mdonça ,You can use following formula to calculate optimal count of partitions based on size of input data and target partition size:Input Stage Data 300GBTarget Size = 200MBOptimal Count of Partitions = 300,000 MB / 200 = 1500 partition...
You can define widgets in your notebook in the following way (see screenshot). When databricks workflow is running it will pass all required parameters and you can use code you've provided to read these values key1=dbutils.widgets.get("Key1").But, le...
