Databricks

Michael42 · 07-20-2023

I found a scenario where the workflow UI seems to not list all the jobs that ran.The objective is to keep trying to run a job, using the job rest api, when the service returns a status of "skipped" while another run is currently running, until the jo...

Michael42 · 06-05-2023

My source can only deliver CSV format (pipe delimited).My source has the ability to generate multiple CSV files and transfer them to a single upload folder.All rows must go to the same target bronze delta table.I do not care about the order in which ...

Michael42 · 11-11-2022

Input Data:One batch load of a daily dataset, roughly 10 million items a day of transactions.Another daily batch load of roughly the same size.Each row in one dataset should have a corresponding row in the other dataset.Problem to solve:The problem i...

Michael42 · 12-19-2023

I recently had this problem on Azure. Created a SQL Warehouse with a 20 minute auto stop setting. When I saw a $1500 bill on my cc in December, I checked on Azure and saw that the SQL Warehouse was up and running since October 8th. I tried to shut it...

Michael42 · 07-21-2023

I resolved this issue as a client side coding error and withdraw the question.

Michael42 · 12-18-2022

I'd am looking at this as well and would like to understand my options here.

Michael42 · 11-15-2022

Thank you Lennart,This was helpful.

Databricks

User Stats

User Activity

Workflow UI not showing all jobs that were run

Want to load a high volume of CSV rows in the fastest way possible (in excess of 5 billion rows). I want the best approach, in terms of speed, for loading into the bronze table.

Would like to start a discussion regarding techniques for joining two relatively large tables of roughly equal size on a daily basis. I realize this may be a bit of a conundrum with databricks, but review the details.

Re: Databricks SQL Warehouse did not auto stop after specified 90 minute interval - why not?

Re: Workflow UI not showing all jobs that were run

Re: ignoreDeletes' option with Delta Live Table streaming source

Re: Would like to start a discussion regarding techniques for joining two relatively large tables of roughly equal size on a daily basis. I realize this may be a bit of a conundrum with databricks, but review the details.