Databricks Community

Hi everyone!We are reading json files from cross-account S3 bucket using Autoloader and decided to switch from directory listing mode to files notification mode.We have set up all permissions mentioned here in our IAM role. But now the pipeline is fa...

Hi everyone!I am trying to setup Autoloader to read json file with specific name from all subfolders under the path except one.Could someone advice how this can be achieved? For example, I need to read from .../*/specific_name.json, but ignore test f...

Hi everyone!I want to use in-memory cached views in a merge into operation, but I am not entirely sure if the exactly saved in-memory view is used in this operation or not.So, suppose I have a table named table_1 and a cached view named cached_view_1...

Hi!We have job, that runs every hour. It extracts data from the API and saves to the databricks table.Sometimes job fails with error "org.apache.spark.SparkException". Here is the full error:An error occurred while calling o7353.saveAsTable. : org.ap...

Hi!We want to run query located in another notebook every streaming microbatch.We were trying dbutils.run.notebook but we always get errorContext not valid. If you are calling this outside the main thread, you must set the Notebook context via dbutil...