Databricks

marcuskw

We've been able to setup a Foreign Catalog using the following documentation:https://learn.microsoft.com/en-us/azure/databricks/query-federation/sql-serverHowever the tables that have RLS using a Security Policy appear empty. I imagine that this solu...

marcuskw · 01-10-2024

We aim to reduce the amount of notebooks we create to a minimum and instead make these fairly flexible. Therefore we have a Factory setup that takes in a parameter to varies the logic.However when it comes to Workflows we are forced to create multipl...

marcuskw · 09-08-2023

ConcurrentAppendException requires a good partitioning strategy, here my logic works without fault for "whenMatchedUpdate" and "whenNotMatchedInsert" logic. When using "whenNotMatchedBySourceUpdate" however it seems that the condition doesn't isolate...

marcuskw

Was a bit quick here, found out that the SUSER_NAME() of the query is of course the connection that was setup.So the User/Password defined here:Once I added that same user to the RLS logic I get the correct result.

marcuskw · 02-20-2024

I imagine Databricks would have to alter the schema of their Jobs API to implement a solution where schedule would also be an Id field instead of just Job Id. I imagine it would be possible to have a lookup table and append what parameters were run a...

marcuskw · 02-20-2024

Unfortunatly not, our current solution is to have multiple workflows that run the same notebooks but with varying input parameters.Results in a bit of workflow bloat both in UI and CI/CD process

marcuskw · 01-18-2024

Hi,I don't think this really helps me in this use case. Optimally I was after having 1 workflow with the following settings:Schedule 1:Running every hourjob_parameter = "SAP" to import Class SAP where we get data related to that ERP systemSchedule 2:...

marcuskw · 01-12-2024

Hi We have a Factory logic that looks something like this: class Factory: def __init__(self, job_parameter: str): self.job_parameter = job_parameter def set_objects(self): if self.job_parameter == "A": fr...

Databricks

User Stats

User Activity

Lakehouse Federation for SQL Server and Security Policy

Create a Workflow Schedule with varying Parameters

whenNotMatchedBySourceUpdate ConcurrentAppendException Partition

Re: Lakehouse Federation for SQL Server and Security Policy

Re: Create a Workflow Schedule with varying Parameters

Re: Create a Workflow Schedule with varying Parameters

Re: Create a Workflow Schedule with varying Parameters

Re: Create a Workflow Schedule with varying Parameters