- 2879 Views
- 8 replies
- 7 kudos
Databricks asset bundles and Dashboards - pass parameters depending on bundle target
Hello everyone !Since Databricks Asset Bundles can now be used to deploy dashboards, I'm wondering how to pass parameters so that the queries for the dev dashboard query the dev catalog, and the dashboard in stg query the stg catalog etc.Is there any...
- 2879 Views
- 8 replies
- 7 kudos
- 7 kudos
Has this been added yet ? having dashboards in asset bundles without support for parameterization is borderline broken
- 7 kudos
- 32 Views
- 2 replies
- 0 kudos
Looking for Advice: Robust Backup Strategy for Databricks System Tables
HI,I’m planning to build a backup system for all Databricks system tables (audit, usage, price, history, etc.) to preserve data beyond retention limits. Currently, I’m using Spark Streaming with readStream + writeStream and checkpointing in LakeFlow ...
- 32 Views
- 2 replies
- 0 kudos
- 0 kudos
Greetings @Danish11052000 , here’s a pragmatic way to choose, based on the nature of Databricks system tables and the guarantees you want. Bottom line For ongoing replication to preserve data beyond free retention, a Lakeflow Declarative Pipeline w...
- 0 kudos
- 16 Views
- 1 replies
- 0 kudos
Claude Access to Workspace and Catalog
I have a question, if we have a Claude corporate account are we able to link that directly to the Playground of Databricks? So that we would not have to add files separately that are already available in our workspace or catalog.
- 16 Views
- 1 replies
- 0 kudos
- 0 kudos
@aiohi yes you should be able to access the files available. https://www.databricks.com/blog/anthropic-claude-37-sonnet-now-natively-available-databricks https://support.claude.com/en/articles/12430928-using-databricks-for-data-analysis Docs for your...
- 0 kudos
- 4761 Views
- 3 replies
- 1 kudos
SQL week format issue its not showing result as 01(ww)
Hi Folks,I've requirement to show the week number as ww format. Please see the below codeselect weekofyear(date_add(to_date(current_date, 'yyyyMMdd'), +35)). also plz refre the screen shot for result.
- 4761 Views
- 3 replies
- 1 kudos
- 22 Views
- 1 replies
- 0 kudos
Scaling Declarative Streaming Pipelines for CDC from On-Prem Database to Lakehouse
We have a scenario where we need to mirror thousands of tables from on-premises Db2 databases to an Azure Lakehouse. The goal is to create mirror Delta tables in the Lakehouse.Since LakeFlow Connect currently does not support direct mirroring from on...
- 22 Views
- 1 replies
- 0 kudos
- 0 kudos
Just use https://flink.apache.org
- 0 kudos
- 53 Views
- 2 replies
- 1 kudos
online tables to synced table, why is it creating a different service principal everytime?
Hello!We started to move our online tables to synced_tables. We just couldnt figure out why it is creating a new service principal everytime we ran the same code we use for online tables?try: fe.create_feature_spec(name=feature_spec_name ...
- 53 Views
- 2 replies
- 1 kudos
- 1 kudos
Greetings @hgm251 , here are some things to consider. Things are working as designed: when you create a new Feature Serving or Model Serving endpoint, Databricks automatically provisions a dedicated service principal for that endpoint, and a fresh...
- 1 kudos
- 29 Views
- 1 replies
- 0 kudos
Reading MongoDB collections into an RDD
Hi, for a Spark job which does some custom computation, I need to access data from a MongoDB collection and access the elements as of type Document. The reason for this is, that I want to apply some custom type serialization which is already implemen...
- 29 Views
- 1 replies
- 0 kudos
- 0 kudos
Greeting @Mathias_Peters , here are some suggestions for your consideration. Analysis You're encountering a common challenge when migrating to newer versions of the MongoDB Spark Connector. The architecture changed significantly between versions 2.x ...
- 0 kudos
- 21 Views
- 1 replies
- 0 kudos
Broadcast Join Failure in Streaming: Failed to store executor broadcast in BlockManager
Hi Databricks Community,I’m running a Structured Streaming job in Databricks with foreachBatch writing to a Delta table.Failed to store executor broadcast spark_join_relation_1622863(size = Some(67141632)) in BlockManager with storageLevel=StorageLev...
- 21 Views
- 1 replies
- 0 kudos
- 0 kudos
Greetings @pooja_bhumandla , here are some helpful hints and tips. Diagnosis Your error indicates that a broadcast join operation is attempting to send ~64MB of data to executors, but the BlockManager cannot store it due to memory constraints. This c...
- 0 kudos
- 49 Views
- 4 replies
- 2 kudos
Resolved! [FREE TRIAL] Missing All-Purpose Clusters Access - New Account
Issue Description: I created a new Databricks Free Trial account ("For Work" plan with $400 credits) but I don't have access to All-Purpose Clusters or PySpark compute. My workspace only shows SQL-only features.Current Setup:- Account Email: ronel.ra...
- 49 Views
- 4 replies
- 2 kudos
- 2 kudos
Ah, got it @pabloratache , I did some digging and here is what I found (learned a few things myself). Thanks for the detailed context — this behavior is expected for the current Databricks 14‑day Free Trial (“For Work” plan). What’s happening with ...
- 2 kudos
- 58 Views
- 2 replies
- 0 kudos
Resolved! Auto Loader vs Batch for Large File Loads
Hi everyone,I'm seeing a dramatic difference in processing times between batch and streaming (Auto Loader) approaches for reading about 250,000 files from S3 in Databricks. My goal is to read metadata from these files and register it as a table (even...
- 58 Views
- 2 replies
- 0 kudos
- 0 kudos
Thank you, Anudeep.I plan to tune Auto Loader by increasing the maxFilesPerTrigger parameter to optimize performance. My decision to use Auto Loader is primarily driven by its built-in backup functionality via cloudFiles.cleanSource.moveDestination, ...
- 0 kudos
- 3009 Views
- 1 replies
- 0 kudos
Databricks Jobs Failure Notification to Azure DevOps as incident
Dear all,Has anyone tried sending Databricks Jobs Failure Notification to Azure DevOps as incident? I see webhook as a OOTB destination for jobs. I am thinking to leverage it. But, like to hear any success stories of it or any other smart approaches....
- 3009 Views
- 1 replies
- 0 kudos
- 0 kudos
Yes, there are successful approaches and best practices for sending Databricks Job Failure notifications to Azure DevOps as incidents, primarily by leveraging the webhook feature as an out-of-the-box (OOTB) destination in Databricks Jobs. The workflo...
- 0 kudos
- 137 Views
- 3 replies
- 5 kudos
Resolved! Broken s3 file paths in File Notifications for auto loader
Suddenly at "2025-10-23T14:12:48.409+00:00", coming file paths from file notification queue started to be urlencoded. Hence, our pipeline gets file not found exception. I think something has changed suddenly and broke notification system. Here are th...
- 137 Views
- 3 replies
- 5 kudos
- 5 kudos
Hello @aonurdemir, Could you please re-run your pipeline now and check? This issue should be mitigated now. It is due to a recent internal bug that led to the unexpected handling of file paths with special characters. You should set ignoreMissingFile...
- 5 kudos
- 20 Views
- 1 replies
- 0 kudos
EXCEL_DATA_SOURCE_NOT_ENABLED Excel data source is not enabled in this cluster
I want to read an Excel xlsx file on DBR 17.3. On the Cluster the library dev.mauch:spark-excel_2.13:4.0.0_0.31.2 is installed. V1 Implementation works fine:df = spark.read.format("dev.mauch.spark.excel").schema(schema).load(excel_file) display(df)V2...
- 20 Views
- 1 replies
- 0 kudos
- 0 kudos
If I build the spark-excel library with another short name (example "excelv2"), everything works fine. https://github.com/nightscape/spark-excel/issues/896#issuecomment-3486861693
- 0 kudos
- 219 Views
- 6 replies
- 6 kudos
Reading empty json file in serverless gives error
I ran a databricks notebook to do incremental loads from files in raw layer to bronze layer tables. Today, I encountered a case where the delta file was empty. I tried running it manually on the serverless compute and encountered an error.df = spark....
- 219 Views
- 6 replies
- 6 kudos
- 6 kudos
Hello @Dhruv-22 , Can you share the schema of the df? Do you have a _corrupt_record column in your dataframe? If yes.. where are you getting it from, because you said its an empty file correct?As per the design ,Spark blocks queries that only referen...
- 6 kudos
- 10523 Views
- 7 replies
- 0 kudos
Log files are not getting deleted automatically after logRetentionDuration internal
Hi team Log files are not getting deleted automatically after logRetentionDuration internal from delta log folder and after analysis , I see checkpoint files are not getting created after 10 commits . Below table properties using spark.sql( f""" ...
- 10523 Views
- 7 replies
- 0 kudos
| User | Count |
|---|---|
| 1619 | |
| 790 | |
| 483 | |
| 349 | |
| 287 |
