Databricks

pgruetter · 11-16-2023

Hello allI have a delta table in bronze layer, let's call it BRZ. It contains 25B rows and many duplicates. It has a version 0 and a version 1, nothing else yet. I then create a silver table SLV by running one deduplication batch job. This creates ve...

pgruetter · 11-13-2023

Hi allTo read from a large Delta table, I'm using readStream but with a trigger(availableNow=True) as I only want to run it daily. This worked well for an intial load and then incremental loads after that.At some point though, I received an error fro...

pgruetter · 04-18-2023

Hi allI have a task of type Notebook, source is Git (Azure DevOps). This task runs fine with my user, but if I change the Owner to a service principal, I get the following error:Run result unavailable: run failed with error message Failed to checkout...

pgruetter · 03-27-2023

Hi allI'm struggling to connect PowerBI service to a Databricks SQL Warehouse using a service principal. I'm following mostly this guide.I created a new app registration in the AAD and created a client secret for it.Now I'm particularly struggling wi...

pgruetter · 11-24-2023

Thanks a lot for the details. One point I still don't get is the difference between these two points (and let's forget vacuum for this):If your streaming query ran 2 weeks ago, it will not reprocess all table versions since then.Instead, it will proc...

pgruetter · 11-21-2023

Still hard to say, but it sounds like my assumption is correct. Because your upsert doesn't which records to update, it needs to scan everything. Make sure that it's properly partitioned, you have a z-index and execute an optimize table.

pgruetter · 11-21-2023

Can you give some more details? Are doing merge statements? How big are the tables?For merge statements i.e. the process needs to read the target table to analyze which parquet files need to be rewritten. If you don't have proper partitioning or z-in...

pgruetter · 11-16-2023

Thanks for the confirmation. Not sure I see everything as your text gets truncated, but it basically confirms that it should work.Anyway: It looks like the incremental load is working. The problem here is, that we receive late arriving facts that tou...

pgruetter · 04-19-2023

But the link describes how my personal user is connected to Azure DevOps. Since I want to use a service principal, it must be something else, right ?

Databricks

User Stats

User Activity

Help me understand streaming logic with Delta Tables

Streaming problems after Vaccum

Run Task as Service Principal with Code in Azure DevOps Repo

How to use Service Principal to connect PowerBI to Databrick SQL Warehouse

Re: Streaming problems after Vaccum

Re: Filtering files for query

Re: Filtering files for query

Re: Help me understand streaming logic with Delta Tables

Re: Run Task as Service Principal with Code in Azure DevOps Repo