Databricks Community

I would say that your token should be manually refreshed as mentioned in the following statement in docs: Databricks tools and SDKs that implement the Databricks client unified authentication standard will automatically generate, refresh, and use Dat...

Databricks does not have connectivity to your local network out of the box.You should setup a VNet and VNet peering (and also firewall rules).Connect your Azure Databricks workspace to your on-premises network - Azure Databricks | Microsoft LearnNetw...

We have gone down a different route where we are using SQL for our calculated layer and then a Python notebook for our aggregated layer. It is much easier to roll up data using a Python UDF than it is trying to work out how to do it in SQL.

Data can be added to the rescued data column when types do not match and when implicit casting does not work. However, check to see if the typecasting you're trying to do is supported by Delta Lake's type widening feature, which gives more flexibilit...

Oakhill, we do provide free onboard training. You might be interested in the "Get Started with Data Engineering on Databricks" session.  You can register here: https://www.databricks.com/training/catalog.  When you are searching the catalog of traini...

Based off of your screenshot, a Spark job has started, and 33/34 tasks are completed. This is usually indicative of some kind of skewed processing. Please refer to this documentation for help identifying and resolving skew

Auto Loader accept globs as input, including negative globs. You can use this to exclude a directory as long as the path is known ahead of time

The error message suggests that autoloader's state is being improperly deleted, most likely by a separate process. If your checkpoint exists inside of the root of a delta table, then VACUUM can delete its files. Make sure that you do not store checkp...

Do you have any custom DNS in your set up? If yes, are you aware of any changes being performed on the same pointing to the databases?

What mode are you using? File notification?

thanks for sharingYes , but I am looking other way around...GENIE Space rooms ---> Add the Generated Query ( Visualize) ---? to my AI/BI dashboard.Looks like I am doing the wrong way, the best way to create AI/BI dashboards and Genie use with that. L...

@stevenayers-bge Autoloader is designed to work best with immutable files. If files are mutable (i.e., they can be updated), it is recommended to set cloudFiles.allowOverwrites = true to ensure that the latest version of the file is read. Please refe...