Databricks Community

Hey @HoussemBL You're correct about DLT not support Auto LC. You can assign any columns in the cluster_by but if you set it to auto, it will throw an error complaining about auto not being present in the list of columns.Maybe, altering thee table to ...

Hi @manish_tanwar  how can we work with streamlit apps in databricks , i have a use case where i want to ingest data from different csv files and ingest it into delta tables 

So, it appears that the there is conflicting documentation on this topic.  I checked with our internal documention and what I found was that you CANNOT install JDBC or ODBC drivers on Serverless.  See limitations here: https://docs.databricks.com/aws...

Hello @Vasu_Kumar_T - We've used Bladebridge to convert from Redshift to Databricks. Bladebridge can definetly convert to Spark SQL, not sure about Scala Spark though.

I guess that is a safe bet.Good luck!

may I ask why you use threadpools?  with jobs you can define multiple tasks which do the same.I'm asking because threadpools and spark resource management can intervene with each other.

hi @jeremy98 Fundamentally, changing some design patterns would help save cluster costs and job failures due to cluster crashing.I understand that you're using a always running cluster to run your workflow, I'm not sure about the usecase but I'd sugg...

Thank you all for your answers!I did use dbutils.notebook.run() inside a while-loop at first but ultimately would run into OOM errors, even if I tried writing in a clearing of cache after each iteration. I'm curious @RefactorDuncan, if you don't mind...

Hi Jaclaglez13,How are you doing today?, As per my understanding, yeah, this issue usually happens when Unity Catalog affects function resolution across different tasks in a Workflow. Since your first task runs fine but the second one fails, it’s lik...

I would like to see this too.