Databricks Community

HI, @jonathan-dufaul For such reports, I think it would be appropriate to click on your profile icon in the top-right corner of the workspace and use the "Send Feedback" option. 

Hi @Yaadhudbe, We would need to review your DLT setup, cluster settings and spark processing to better understand the OOM errors and possible suggestions to mitigate the issue. I suggest to file a case with us to conduct a proper investigation.  http...

The AvailableNow trigger processes all available data as a single batch and then stops. This is different from continuous or micro-batch processing where the system continuously checks for new data. When a query starts with the AvailableNow trigger, ...

I would say that your token should be manually refreshed as mentioned in the following statement in docs: Databricks tools and SDKs that implement the Databricks client unified authentication standard will automatically generate, refresh, and use Dat...

Databricks does not have connectivity to your local network out of the box.You should setup a VNet and VNet peering (and also firewall rules).Connect your Azure Databricks workspace to your on-premises network - Azure Databricks | Microsoft LearnNetw...

We have gone down a different route where we are using SQL for our calculated layer and then a Python notebook for our aggregated layer. It is much easier to roll up data using a Python UDF than it is trying to work out how to do it in SQL.

Data can be added to the rescued data column when types do not match and when implicit casting does not work. However, check to see if the typecasting you're trying to do is supported by Delta Lake's type widening feature, which gives more flexibilit...

Oakhill, we do provide free onboard training. You might be interested in the "Get Started with Data Engineering on Databricks" session.  You can register here: https://www.databricks.com/training/catalog.  When you are searching the catalog of traini...

Based off of your screenshot, a Spark job has started, and 33/34 tasks are completed. This is usually indicative of some kind of skewed processing. Please refer to this documentation for help identifying and resolving skew

Auto Loader accept globs as input, including negative globs. You can use this to exclude a directory as long as the path is known ahead of time

The error message suggests that autoloader's state is being improperly deleted, most likely by a separate process. If your checkpoint exists inside of the root of a delta table, then VACUUM can delete its files. Make sure that you do not store checkp...

Do you have any custom DNS in your set up? If yes, are you aware of any changes being performed on the same pointing to the databases?