- 433 Views
- 2 replies
- 1 kudos
Resolved! Migrating from on-premises HDFS to Unity Catalog - Looking for advice on on-prem options
Hi,We’re currently running a Databricks installation with an on-premises HDFS file system. As we’re looking to adopt Unity Catalog, we’ve realized that our current HDFS setup has limited support and compatibility with Unity Catalog.Our requirement: W...
- 433 Views
- 2 replies
- 1 kudos
- 1 kudos
Thanks very much for your detailed response — this is really helpful.You mentioned client cases where organizations have migrated from on-premises HDFS into the Databricks Unity Catalog, I’d love to learn more about those.If possible, could you share...
- 1 kudos
- 354 Views
- 2 replies
- 0 kudos
Looking for Advice: Robust Backup Strategy for Databricks System Tables
HI,I’m planning to build a backup system for all Databricks system tables (audit, usage, price, history, etc.) to preserve data beyond retention limits. Currently, I’m using Spark Streaming with readStream + writeStream and checkpointing in LakeFlow ...
- 354 Views
- 2 replies
- 0 kudos
- 0 kudos
Greetings @Danish11052000 , here’s a pragmatic way to choose, based on the nature of Databricks system tables and the guarantees you want. Bottom line For ongoing replication to preserve data beyond free retention, a Lakeflow Declarative Pipeline w...
- 0 kudos
- 211 Views
- 1 replies
- 0 kudos
Resolved! Claude Access to Workspace and Catalog
I have a question, if we have a Claude corporate account are we able to link that directly to the Playground of Databricks? So that we would not have to add files separately that are already available in our workspace or catalog.
- 211 Views
- 1 replies
- 0 kudos
- 0 kudos
@aiohi yes you should be able to access the files available. https://www.databricks.com/blog/anthropic-claude-37-sonnet-now-natively-available-databricks https://support.claude.com/en/articles/12430928-using-databricks-for-data-analysis Docs for your...
- 0 kudos
- 204 Views
- 1 replies
- 1 kudos
Resolved! Reading MongoDB collections into an RDD
Hi, for a Spark job which does some custom computation, I need to access data from a MongoDB collection and access the elements as of type Document. The reason for this is, that I want to apply some custom type serialization which is already implemen...
- 204 Views
- 1 replies
- 1 kudos
- 1 kudos
Greeting @Mathias_Peters , here are some suggestions for your consideration. Analysis You're encountering a common challenge when migrating to newer versions of the MongoDB Spark Connector. The architecture changed significantly between versions 2.x ...
- 1 kudos
- 293 Views
- 1 replies
- 1 kudos
Resolved! Broadcast Join Failure in Streaming: Failed to store executor broadcast in BlockManager
Hi Databricks Community,I’m running a Structured Streaming job in Databricks with foreachBatch writing to a Delta table.Failed to store executor broadcast spark_join_relation_1622863(size = Some(67141632)) in BlockManager with storageLevel=StorageLev...
- 293 Views
- 1 replies
- 1 kudos
- 1 kudos
Greetings @pooja_bhumandla , here are some helpful hints and tips. Diagnosis Your error indicates that a broadcast join operation is attempting to send ~64MB of data to executors, but the BlockManager cannot store it due to memory constraints. This c...
- 1 kudos
- 358 Views
- 4 replies
- 5 kudos
Resolved! [FREE TRIAL] Missing All-Purpose Clusters Access - New Account
Issue Description: I created a new Databricks Free Trial account ("For Work" plan with $400 credits) but I don't have access to All-Purpose Clusters or PySpark compute. My workspace only shows SQL-only features.Current Setup:- Account Email: ronel.ra...
- 358 Views
- 4 replies
- 5 kudos
- 5 kudos
Ah, got it @pabloratache , I did some digging and here is what I found (learned a few things myself). Thanks for the detailed context — this behavior is expected for the current Databricks 14‑day Free Trial (“For Work” plan). What’s happening with ...
- 5 kudos
- 365 Views
- 2 replies
- 0 kudos
Resolved! Auto Loader vs Batch for Large File Loads
Hi everyone,I'm seeing a dramatic difference in processing times between batch and streaming (Auto Loader) approaches for reading about 250,000 files from S3 in Databricks. My goal is to read metadata from these files and register it as a table (even...
- 365 Views
- 2 replies
- 0 kudos
- 0 kudos
Thank you, Anudeep.I plan to tune Auto Loader by increasing the maxFilesPerTrigger parameter to optimize performance. My decision to use Auto Loader is primarily driven by its built-in backup functionality via cloudFiles.cleanSource.moveDestination, ...
- 0 kudos
- 3313 Views
- 1 replies
- 0 kudos
Databricks Jobs Failure Notification to Azure DevOps as incident
Dear all,Has anyone tried sending Databricks Jobs Failure Notification to Azure DevOps as incident? I see webhook as a OOTB destination for jobs. I am thinking to leverage it. But, like to hear any success stories of it or any other smart approaches....
- 3313 Views
- 1 replies
- 0 kudos
- 0 kudos
Yes, there are successful approaches and best practices for sending Databricks Job Failure notifications to Azure DevOps as incidents, primarily by leveraging the webhook feature as an out-of-the-box (OOTB) destination in Databricks Jobs. The workflo...
- 0 kudos
- 359 Views
- 3 replies
- 5 kudos
Resolved! Broken s3 file paths in File Notifications for auto loader
Suddenly at "2025-10-23T14:12:48.409+00:00", coming file paths from file notification queue started to be urlencoded. Hence, our pipeline gets file not found exception. I think something has changed suddenly and broke notification system. Here are th...
- 359 Views
- 3 replies
- 5 kudos
- 5 kudos
Hello @aonurdemir, Could you please re-run your pipeline now and check? This issue should be mitigated now. It is due to a recent internal bug that led to the unexpected handling of file paths with special characters. You should set ignoreMissingFile...
- 5 kudos
- 11007 Views
- 7 replies
- 0 kudos
Log files are not getting deleted automatically after logRetentionDuration internal
Hi team Log files are not getting deleted automatically after logRetentionDuration internal from delta log folder and after analysis , I see checkpoint files are not getting created after 10 commits . Below table properties using spark.sql( f""" ...
- 11007 Views
- 7 replies
- 0 kudos
- 1056 Views
- 2 replies
- 1 kudos
Modifying size of /var/lib/lxc
Good morning,When running a library (sparkling water) for a very large dataset, I've noticed that during an export procedure the /var/lib/lxc storage is being used. Since the storage seems to be at a static 130GB of memory, this is a problem because ...
- 1056 Views
- 2 replies
- 1 kudos
- 1 kudos
Unfortunately this is a setting that cannot be increased on customer side
- 1 kudos
- 6771 Views
- 5 replies
- 0 kudos
Which is better - Azure Databricks or GCP Databricks?
Which cloud hosting environment is best to use for Databricks? My question pins down to the fact that there must be some difference between the latency, throughput, result consistency & reproducibility between different cloud hosting environments of ...
- 6771 Views
- 5 replies
- 0 kudos
- 0 kudos
@VikasSinha Databricks is not stable regardless of the cloud, jobs and clusters keep crashing. Use Polars or Duckdb instead.
- 0 kudos
- 222 Views
- 1 replies
- 1 kudos
Resolved! Missing warehouse id/metadata for the system compute warehouse table
I ran the following queries for a specific warehouse_id = '54a93d2138433216' SELECT * FROM system.billing.usage WHERE usage_metadata.warehouse_id = '54a93d2138433216';SELECT * FROM system.compute.warehouse_events WHERE warehouse_id = '54a93d213843321...
- 222 Views
- 1 replies
- 1 kudos
- 1 kudos
@Danish11052000 The issue can happen when the warehouse created is very old (let's say last year), then you may not see the details in system table. If they were deleted before the table was created, you'll not see it.
- 1 kudos
- 768 Views
- 1 replies
- 2 kudos
Resolved! Webinars
Hi! My colleagues and I at Unifeye are hosting a series of regular webinars focused on Databricks content. In November, we’re running four sessions covering Geospatial, Governance, AI, and Delta Sharing, featuring Databricks architects as guest speak...
- 768 Views
- 1 replies
- 2 kudos
- 2 kudos
Hello @bianca_unifeye! Thanks for sharing! This looks like a great initiative.For better visibility and engagement, please go ahead and post about the Webinar series directly in the Community as a post. Members can then register and join through your...
- 2 kudos
- 169 Views
- 0 replies
- 1 kudos
Webinar: Geospatial Data Ingestion and Manipulation on Databricks
Geospatial Data Meets Databricks + Felt: Turning Coordinates into Business InsightMost organisations capture huge volumes of spatial data — addresses, coordinates, routes, catchments — but struggle to operationalise it at scale. Traditional GIS tool...
- 169 Views
- 0 replies
- 1 kudos
| User | Count |
|---|---|
| 1629 | |
| 790 | |
| 511 | |
| 349 | |
| 287 |
