We've been able to setup a Foreign Catalog using the following documentation:https://learn.microsoft.com/en-us/azure/databricks/query-federation/sql-serverHowever the tables that have RLS using a Security Policy appear empty. I imagine that this solu...
We aim to reduce the amount of notebooks we create to a minimum and instead make these fairly flexible. Therefore we have a Factory setup that takes in a parameter to varies the logic.However when it comes to Workflows we are forced to create multipl...
ConcurrentAppendException requires a good partitioning strategy, here my logic works without fault for "whenMatchedUpdate" and "whenNotMatchedInsert" logic. When using "whenNotMatchedBySourceUpdate" however it seems that the condition doesn't isolate...
What I wanted to achieve was a dynamic schema application based on what file was picked up.So I implement 1 autoloader task to collect files from a specific path "source":- source/employees/0001.csv- source/holiday/0001.csvIf the path of the file was...
You are able to provide filter options to select only relevant files:https://docs.databricks.com/en/ingestion/cloud-object-storage/auto-loader/patterns.html#filtering-directories-or-files-using-glob-patterns
If it helps anyone else I found this article that described a few limitations:https://community.databricks.com/t5/technical-blog/how-not-to-build-an-execute-immediate-demo/ba-p/82167
Doesn't the docs say it is supported? (DML)"A column or view referenced in a query. This includes queries embedded in a DDL or DML statement.When using the identifier clause it may not be embedded within an identifier."Here I'm referencing a column i...
Hi,I wrote it in the bottom of my post that I know that updating a Temp table isn't allowed.I used it as a quick example to illustrate the error message, but you get the same error message when updating a Delta Lake table.The problem is the IDENTIFIE...
