Databricks Community is an open-source platform for data enthusiasts and professionals to discuss, share insights, and collaborate on everything related to Databricks. Members can ask questions, share knowledge, and support each other in an environment that ensures respectful interactions.
I have a stream to static join, but some of the rows in the static table arrive later than the linked rows in the stream. What is the default behaviour if a record in the stream hasn't joined a ...
Hello, I create a job with one pipeline task through DAB. Now I want to provide a variable to it, but it is dynamic based on the target environment. As pipeline tasks do not support widgets, how can...
Hello, I am using databricks autoloader with managedfileevents turned on and include existing files. I want to understand if there is a way of increasing the speed of the initial listing of the fi...
The following serverless query has an aggregated task time of 1.94h, but each of the two executions runs in an aggregated time of just a few minutes. How is one supposed to make sense of that?
Hi everyone. I have a very wide table (600 columns) on an Oracle database that I need to extract to my data lake. The table has approximately 700K rows. I am currently trying to extract and load...
I have a question regarding Databricks Dashboards. I would like to fix the filter area within the dashboard so that it remains visible at all times, even when scrolling up or down. Is there a way to...
I am defining a job using Databricks Asset Bundles (DABs) as follows: YAML resources:
jobs:
job_name:
...
schedule:
{{ if eq ${var.env} "prd" }}
pause_s...
I am currently using Databricks Asset Bundles (DABs) to deploy and manage dashboard resources. While I can manually add a schedule to a dashboard via the Databricks console, I would like to reflect t...
Anyone know where to see any logs related to Lakebase/Postgres? I have a Tableau Prep flow that is failing but the error is not clear and I'm trying to find out what the database is capturing.
Hey I'm trying to add mode: development to my "Development" target (which is default) but it does not seem to be working as I expected. Here is what my targets file looks like: I'm deploying wit...