Databricks

Hubert_Dudek1

Streamline Data Modeling Normalization with Databricks Delta Live Tables in Just a Few Steps:- Use the "Apply changes" function to populate tables with slowly changing dimensions using auto-increment IDs.- Register SQL mapping functions to associate ...

Hubert_Dudek1

With the introduction of the Unity Catalog in databricks, many of us have become familiar with creating catalogs. However, did you know that the Unity Catalog also allows you to create foreign catalogs? You can register databases from the following s...

Hubert_Dudek1

Always set a timeout for your jobs! It not only safeguards against unforeseen hang-ups but also optimizes resource utilization. Equally essential is to consider having a threshold warning. This can alert you before a potential failure, allowing proac...

Hubert_Dudek1

Did you know the default timeout setting for SQL #databricks Warehouse is two days?The default timeout can be too long for most use cases. You can easily change this for your session or in the general SQL warehouse configuration.

Hubert_Dudek1

#databricks #spark 3.3 has introduced a simple yet powerful isEmpty() function for DataFrames. Gone are the days of using count() to check for empty DataFrames — now it'**bleep** as easy as calling df.isEmpty().

Hubert_Dudek1 · 08-04-2023

Thank you for information

Hubert_Dudek1 · 08-03-2023

There is API 2.1 available for jobs and here is better documentation https://docs.databricks.com/api/workspace/jobs/list

Hubert_Dudek1 · 08-03-2023

- In the EventHub, you can preview the event hub job using Azure Analitycs, so please first check are all records there- Please set in Databricks that it is saved directly to the bronze delta table without performing any aggregation, just 1 to 1, and...

Hubert_Dudek1 · 08-03-2023

Is that program even still exists?

Hubert_Dudek1 · 08-03-2023

It always takes a few minutes, even 10, before the machine is terminated in Azure. It is better to set a pool of machines in data bricks and use them from there so we will not be keeping Zombie machines.

Databricks

User Stats

User Activity

Streaming Data Modeling Normalization with Databricks Delta Live Tables

Foreign catalogs

Workflow timeout

Optimizing SQL Databricks Warehouse Timeout Settings

Checking that spark dataframe is empty

Re: Databricks Rewards Portal and the points credited in it

Re: Response for list job runs API request doesn't have next/prev_page_token field

Re: Data getting missed while reading from azure event hub using spark streaming

Re: Databricks Rewards Portal and the points credited in it

Re: Multiple orphan vms in managed resource group after starting and terminating my personal cluster