Databricks Community is the official forum for data engineers, data scientists, data enthusiasts and AI practitioners to ask questions, share knowledge, and collaborate on everything related to Databricks. Members can find answers, share best practices, and support each other in an environment that ensures respectful interactions.
...s disabled. The Automatic CDF workspace preview is enabled (says GA now) The producer pipeline is configured with the PREVIEW channel and pipelines.externalMetadata.enabled=true. I recreated and r...
We have six years of Parquet files organised under yyyy/mm/dd/, with around 100 table files in each daily folder. We want to process these 100 tables using Auto Loader in a Declarative Pipeline with ...
Hi everyone, I'm working on a client use case where they want to visualize data quality together with data lineage. My current approach is to combine: Unity Catalog Data Lineage for upstream/down...
Using read_files to read Avro-formatted blobs from an external location, I get the error: > Malformed records are detected in record parsing: Event 0. Parse Mode: FAILFAST. To process malformed r...
Databricks Jobs `for_each_task` accepts exactly one nested task. For workloads that run the *same multi-step pipeline over many datasets* — a very common metadata-driven pattern — this forces a cho...
Hi Everyone, There is a need for a lakehouse front-end tool at my company that operates like an excel spreadsheet but reads/writes to a lakehouse table. This would replace actual spreadsheets that ar...
I have identified a serious issue in Genie Agents that requires attention from the Databricks Product Team. When a catalog is renamed, the tables that were previously configured and used within the ...
I’m a Databricks Academy Labs Subscription customer currently working through the self-paced learning plan “Deploy Workloads with Lakeflow Jobs” in preparation for the Databricks DE Associate certifi...
Hello everyone, I am new in Databricks, and currently I am in the 8th lecture of the "Get started with data engineering" module. I came across this problem: [STREAMING_TABLE_OPERATION_NOT...
Hi everyone, I’m trying to create an Azure Key Vault-backed secret scope in a Databricks Premium workspace, but I keep getting this error: Fetch request failed due expired user session Set...