- 2308 Views
- 4 replies
- 5 kudos
Resolved! [FREE TRIAL] Missing All-Purpose Clusters Access - New Account
Issue Description: I created a new Databricks Free Trial account ("For Work" plan with $400 credits) but I don't have access to All-Purpose Clusters or PySpark compute. My workspace only shows SQL-only features.Current Setup:- Account Email: ronel.ra...
- 2308 Views
- 4 replies
- 5 kudos
- 5 kudos
Ah, got it @pabloratache , I did some digging and here is what I found (learned a few things myself). Thanks for the detailed context — this behavior is expected for the current Databricks 14‑day Free Trial (“For Work” plan). What’s happening with ...
- 5 kudos
- 3336 Views
- 2 replies
- 0 kudos
Resolved! Auto Loader vs Batch for Large File Loads
Hi everyone,I'm seeing a dramatic difference in processing times between batch and streaming (Auto Loader) approaches for reading about 250,000 files from S3 in Databricks. My goal is to read metadata from these files and register it as a table (even...
- 3336 Views
- 2 replies
- 0 kudos
- 0 kudos
Thank you, Anudeep.I plan to tune Auto Loader by increasing the maxFilesPerTrigger parameter to optimize performance. My decision to use Auto Loader is primarily driven by its built-in backup functionality via cloudFiles.cleanSource.moveDestination, ...
- 0 kudos
- 4757 Views
- 1 replies
- 0 kudos
Databricks Jobs Failure Notification to Azure DevOps as incident
Dear all,Has anyone tried sending Databricks Jobs Failure Notification to Azure DevOps as incident? I see webhook as a OOTB destination for jobs. I am thinking to leverage it. But, like to hear any success stories of it or any other smart approaches....
- 4757 Views
- 1 replies
- 0 kudos
- 0 kudos
Yes, there are successful approaches and best practices for sending Databricks Job Failure notifications to Azure DevOps as incidents, primarily by leveraging the webhook feature as an out-of-the-box (OOTB) destination in Databricks Jobs. The workflo...
- 0 kudos
- 1670 Views
- 3 replies
- 5 kudos
Resolved! Broken s3 file paths in File Notifications for auto loader
Suddenly at "2025-10-23T14:12:48.409+00:00", coming file paths from file notification queue started to be urlencoded. Hence, our pipeline gets file not found exception. I think something has changed suddenly and broke notification system. Here are th...
- 1670 Views
- 3 replies
- 5 kudos
- 5 kudos
Hello @aonurdemir, Could you please re-run your pipeline now and check? This issue should be mitigated now. It is due to a recent internal bug that led to the unexpected handling of file paths with special characters. You should set ignoreMissingFile...
- 5 kudos
- 1852 Views
- 2 replies
- 1 kudos
Modifying size of /var/lib/lxc
Good morning,When running a library (sparkling water) for a very large dataset, I've noticed that during an export procedure the /var/lib/lxc storage is being used. Since the storage seems to be at a static 130GB of memory, this is a problem because ...
- 1852 Views
- 2 replies
- 1 kudos
- 1 kudos
Unfortunately this is a setting that cannot be increased on customer side
- 1 kudos
- 10650 Views
- 5 replies
- 0 kudos
Which is better - Azure Databricks or GCP Databricks?
Which cloud hosting environment is best to use for Databricks? My question pins down to the fact that there must be some difference between the latency, throughput, result consistency & reproducibility between different cloud hosting environments of ...
- 10650 Views
- 5 replies
- 0 kudos
- 0 kudos
@VikasSinha Databricks is not stable regardless of the cloud, jobs and clusters keep crashing. Use Polars or Duckdb instead.
- 0 kudos
- 864 Views
- 1 replies
- 1 kudos
Resolved! Missing warehouse id/metadata for the system compute warehouse table
I ran the following queries for a specific warehouse_id = '54a93d2138433216' SELECT * FROM system.billing.usage WHERE usage_metadata.warehouse_id = '54a93d2138433216';SELECT * FROM system.compute.warehouse_events WHERE warehouse_id = '54a93d213843321...
- 864 Views
- 1 replies
- 1 kudos
- 1 kudos
@Danish11052000 The issue can happen when the warehouse created is very old (let's say last year), then you may not see the details in system table. If they were deleted before the table was created, you'll not see it.
- 1 kudos
- 1316 Views
- 1 replies
- 2 kudos
Resolved! Webinars
Hi! My colleagues and I at Unifeye are hosting a series of regular webinars focused on Databricks content. In November, we’re running four sessions covering Geospatial, Governance, AI, and Delta Sharing, featuring Databricks architects as guest speak...
- 1316 Views
- 1 replies
- 2 kudos
- 2 kudos
Hello @bianca_unifeye! Thanks for sharing! This looks like a great initiative.For better visibility and engagement, please go ahead and post about the Webinar series directly in the Community as a post. Members can then register and join through your...
- 2 kudos
- 719 Views
- 0 replies
- 1 kudos
Webinar: Geospatial Data Ingestion and Manipulation on Databricks
Geospatial Data Meets Databricks + Felt: Turning Coordinates into Business InsightMost organisations capture huge volumes of spatial data — addresses, coordinates, routes, catchments — but struggle to operationalise it at scale. Traditional GIS tool...
- 719 Views
- 0 replies
- 1 kudos
- 4170 Views
- 5 replies
- 3 kudos
Resolved! in data bricks free edition Serverless compute is not working
I recently logged into the Databricks free edition, but the serverless compute is not working. I'm receiving the error: 'An error occurred while trying to attach serverless compute. Please try again or contact support.'"
- 4170 Views
- 5 replies
- 3 kudos
- 3 kudos
Hi @Abdul_Alikhan ,I experienced the same yesterday when I imported some of my notebooks. I noticed that this error only appeared for imported notebooks because the environment version was 1 by default. Changing the environment version to 2 solved th...
- 3 kudos
- 2488 Views
- 1 replies
- 1 kudos
Resolved! Databricks Streaming: Recommended Cluster Types and Best Practices
Hi Community, I recently built some streaming pipelines (Autoloader-based) that extract JSON data from the Data Lake and, after parsing and logging, dump it into the Delta Lake bronze layer. Since these are streaming pipelines, they are supposed to r...
- 2488 Views
- 1 replies
- 1 kudos
- 1 kudos
When running streaming pipelines, the key is to design for stability and isolation, not to rely on restart jobs.The first thing to do is run your streams on Jobs Compute, not All-Purpose clusters. If available, use Serverless Jobs. Each pipeline shou...
- 1 kudos
- 3290 Views
- 2 replies
- 0 kudos
Resolved! I do not have any Spark jobs running, but my cluster is not getting auto-terminated.
The cluster is Idle and there are no Spark jobs running on the Spark UI. Still I see my cluster is active and not getting terminated.
- 3290 Views
- 2 replies
- 0 kudos
- 0 kudos
Databricks cluster is treated as active if there are any spark or non-Spark operations running on the cluster. Even though there are no Spark jobs running on the cluster, it's possible to have some driver-specific application code running marking th...
- 0 kudos
- 1014 Views
- 2 replies
- 2 kudos
Resolved! Course - Introduction to Apache Spark
Hi,In the course Introduction to Apache Spark; according to Apache Spark Runtime Architecture; Page 6 of 15. It says that :The cluster manager allocates resources and assigns tasks......Workers perform tasks assigned by the driverCan you help me plea...
- 1014 Views
- 2 replies
- 2 kudos
- 2 kudos
Hi @fundat Perhaps the picture is useful here:Give this blog a read, I think this will answer some of your questions: https://medium.com/@knoldus/understanding-the-working-of-spark-driver-and-executor-4fec0e669399 .All the best,BS
- 2 kudos
- 9522 Views
- 12 replies
- 1 kudos
Multiple concurrent jobs using interactive cluster
Hi All,I have notebook in Databricks. This notebook is executed from azure datafactory pipeline having a databricks notebook activity with linkedservice connected to an interactive cluster.When multiple concurrent runs of this pipeline are created, I...
- 9522 Views
- 12 replies
- 1 kudos
- 1 kudos
Greetings @jigar191089 , I did some digging and here are some ideas to think about. This smells like a shared-state/import-path issue on an interactive cluster under concurrency. What likely happened Your notebook imports Python modules from /dbf...
- 1 kudos
- 2101 Views
- 2 replies
- 1 kudos
Resolved! DLT | Communication lost with driver | Cluster was not reachable for 120 seconds
Hey Community, I'm facing this error, It says that "com.databricks.pipelines.common.errors.deployment.DeploymentException: Communication lost with driver. Cluster 1030-205818-yu28ft9s was not reachable for 120 seconds" This issue occurred in producti...
- 2101 Views
- 2 replies
- 1 kudos
- 1 kudos
This is actually a known intermittent issue in Databricks, particularly with streaming or Delta Live Tables (DLT) pipelines.This isn’t a logical failure in your code — it’s an infrastructure-level timeout between the Databricks control plane and the ...
- 1 kudos
| User | Count |
|---|---|
| 1645 | |
| 793 | |
| 584 | |
| 349 | |
| 287 |
