Databricks Community

Advancing Analytics have a good (although contrarian) take on the medallion architecture: Behind the Hype - The Medallion Architecture Doesn't Work - YouTubeI found that vid very helpful.

Hey, you can use the pricing calculator here: https://www.databricks.com/product/pricing/product-pricing/instance-types

you need to set SET TBLPROPERTIES ('delta.checkpointRetentionDuration' = '30 days',)

Another point I have realised, is that the task and the parent notebook (which then calls the child notebook that runs the auto loader part) does not fail if the schema-changed failure occurs during the auto loader process.  It's the child notebook a...

Worked it out with Microsoft.-> only works with external volumes, not managed.https://learn.microsoft.com/en-us/azure/databricks/workflows/jobs/file-arrival-triggers 

Hi!I have same issue with insert_overwrite on Databricks with SQL Warehouse. Do you have any solution or updates? Or is it still not supported by Databricks? 

Hey @ShlomoSQM, looks like @shan_chandra suggested a feasible solution, just to add a little more context this is how you can achieve the same if you have a column that can help you identify what is type1 and type 2file_type1_stream = readStream.opti...

You can use:${param_catalog}.schema.tablename.Pass actual value in the notebook through a job param "param_catalog" or widget utils through text called "param_catalog"