Databricks Community

ac0 · 05-09-2024

Is it more performant to run optimize table commands on a serverless SQL warehouse or in a job or all-purpose compute cluster? I would presume a serverless warehouse would be faster, but I don't know how to test this.

ac0 · 03-31-2024

Does anyone have documentation on what is actually occurring in each of these steps?Creating update Waiting for resourcesInitializingSetting up tablesRendering graphFor example, what is the difference between initializing and setting up tables? I am ...

ac0 · 03-20-2024

Currently my Databricks Metastore is in the the same location as the data for my production catalog. We are moving the data to a separate storage account. In advance of this, I'm curious if there is a way to determine the size of the metastore itself...

ac0 · 03-08-2024

I have an Azure Storage Data Table that I would like to update based on records that were just streamed into a Delta Live Table. Below is example code:@Dlt.create_table( comment="comment", table_properties={ "pipelines.autoOptimize.managed": ...

ac0 · 03-05-2024

Running almost any notebook with a merge statement in Databricks with DBR 14.3 I get the following error and the notebook exists:"Fatal error: The Python kernel is unresponsive."I would provide more code, but like I said, it is pretty much anything w...

ac0 · 03-29-2024

Pitching in here since i had the same problem: the issue you can't delete from a CTE; you need to delete from the actual table which can be joined to the CTE.

ac0 · 03-14-2024

Bumping this thread as well.

ac0 · 03-08-2024

@Kaniz , Are you suggesting that the first time a pipeline runs, the below decorator should be used: @dlt.create_table And the once the pipeline runs, the notebook should be edited to use this decorator: @dlt.table

ac0 · 03-06-2024

Thanks @Ayushi_Suthar. I will try this and report back. Is "spark.databricks.driver.python.pythonHealthCheckTimeoutSec" a new configuration setting? Literally nothing on Google or Stackoverflow is returned when I search by either that full name or ev...

ac0 · 01-18-2024

I was able to accomplish this by creating a Cluster Policy that put in place the scripts, config settings, and environment variables I needed.

Databricks Community

User Stats

User Activity

Is it more performant to run optimize table commands on a serverless SQL warehouse or elsewhere?

Delta Live Table pipeline steps explanation

Get size of metastore specifically

Can I have additional logic in a DLT notebook that is unrelated to directly creating DLTs?

"Fatal error: The Python kernel is unresponsive." DBR 14.3

Re: Permission Error When Running DELETE FROM

Re: How to move a metastore to a new Storage Account in unity catalog?

Re: Difference @dlt.table and @dlt.create_table decorator

Re: "Fatal error: The Python kernel is unresponsive." DBR 14.3

Re: Setting environment variables to use in a SQL Delta Live Table Pipeline