Databricks

I am exploring the use of the "file arrival" trigger for a workflow for a use case I am working on.  I understand from the documentation that it checks every minute for new files in an external location, then initiates the workflow when it detects a ...

I have a parquet file that I am trying to write to a delta table:df.writeStream  .format("delta")  .option("checkpointLocation", f"{targetPath}/delta/{tableName}/__checkpoints")  .trigger(once=True)  .foreachBatch(processTable)  .outputMode("append")...

We observed the following error in a notebook which was running from a Databricks workflow: ModuleNotFoundError: No module named '<python package>'The error message speaks for itself - it obviously couldn't find the python package.  What is peculiar ...

I am trying to run the following chunk of code in the cell of a Databricks notebook (using Databricks runtime 14.3 LTS, Apache spark 3.5.0, scala 2.12): spark.sql("CREATE OR REPLACE table sample_catalog.sample_schema.sample_table_tmp AS SELECT * FROM...

I have an SCD Type 1 delta table (target) for which I am trying to figure out how to facilitate insert, updates, and deletes.  This table is sourced by multiple delta tables, with an SCD Type 2 structure, which are joined together to create the targe...