- 141 Views
- 1 replies
- 1 kudos
Resolved! Missing warehouse id/metadata for the system compute warehouse table
I ran the following queries for a specific warehouse_id = '54a93d2138433216' SELECT * FROM system.billing.usage WHERE usage_metadata.warehouse_id = '54a93d2138433216';SELECT * FROM system.compute.warehouse_events WHERE warehouse_id = '54a93d213843321...
- 141 Views
- 1 replies
- 1 kudos
- 1 kudos
@Danish11052000 The issue can happen when the warehouse created is very old (let's say last year), then you may not see the details in system table. If they were deleted before the table was created, you'll not see it.
- 1 kudos
- 663 Views
- 1 replies
- 2 kudos
Resolved! Webinars
Hi! My colleagues and I at Unifeye are hosting a series of regular webinars focused on Databricks content. In November, we’re running four sessions covering Geospatial, Governance, AI, and Delta Sharing, featuring Databricks architects as guest speak...
- 663 Views
- 1 replies
- 2 kudos
- 2 kudos
Hello @bianca_unifeye! Thanks for sharing! This looks like a great initiative.For better visibility and engagement, please go ahead and post about the Webinar series directly in the Community as a post. Members can then register and join through your...
- 2 kudos
- 107 Views
- 0 replies
- 1 kudos
Webinar: Geospatial Data Ingestion and Manipulation on Databricks
Geospatial Data Meets Databricks + Felt: Turning Coordinates into Business InsightMost organisations capture huge volumes of spatial data — addresses, coordinates, routes, catchments — but struggle to operationalise it at scale. Traditional GIS tool...
- 107 Views
- 0 replies
- 1 kudos
- 1184 Views
- 5 replies
- 3 kudos
Resolved! in data bricks free edition Serverless compute is not working
I recently logged into the Databricks free edition, but the serverless compute is not working. I'm receiving the error: 'An error occurred while trying to attach serverless compute. Please try again or contact support.'"
- 1184 Views
- 5 replies
- 3 kudos
- 3 kudos
Hi @Abdul_Alikhan ,I experienced the same yesterday when I imported some of my notebooks. I noticed that this error only appeared for imported notebooks because the environment version was 1 by default. Changing the environment version to 2 solved th...
- 3 kudos
- 136 Views
- 1 replies
- 1 kudos
Resolved! Databricks Streaming: Recommended Cluster Types and Best Practices
Hi Community, I recently built some streaming pipelines (Autoloader-based) that extract JSON data from the Data Lake and, after parsing and logging, dump it into the Delta Lake bronze layer. Since these are streaming pipelines, they are supposed to r...
- 136 Views
- 1 replies
- 1 kudos
- 1 kudos
When running streaming pipelines, the key is to design for stability and isolation, not to rely on restart jobs.The first thing to do is run your streams on Jobs Compute, not All-Purpose clusters. If available, use Serverless Jobs. Each pipeline shou...
- 1 kudos
- 85 Views
- 0 replies
- 0 kudos
Optimizing SAP S/4HANA Data Integration into Databricks for Real-Time Analytics
Hey everyone,I’ve been building a data pipeline to bring SAP S/4HANA data into Databricks for real-time analytics and reporting. The setup relies on Delta Lake for storage, and while batch ingestion works fine, I’m running into issues with latency an...
- 85 Views
- 0 replies
- 0 kudos
- 2486 Views
- 2 replies
- 0 kudos
Resolved! I do not have any Spark jobs running, but my cluster is not getting auto-terminated.
The cluster is Idle and there are no Spark jobs running on the Spark UI. Still I see my cluster is active and not getting terminated.
- 2486 Views
- 2 replies
- 0 kudos
- 0 kudos
Databricks cluster is treated as active if there are any spark or non-Spark operations running on the cluster. Even though there are no Spark jobs running on the cluster, it's possible to have some driver-specific application code running marking th...
- 0 kudos
- 211 Views
- 2 replies
- 2 kudos
Resolved! Course - Introduction to Apache Spark
Hi,In the course Introduction to Apache Spark; according to Apache Spark Runtime Architecture; Page 6 of 15. It says that :The cluster manager allocates resources and assigns tasks......Workers perform tasks assigned by the driverCan you help me plea...
- 211 Views
- 2 replies
- 2 kudos
- 2 kudos
Hi @fundat Perhaps the picture is useful here:Give this blog a read, I think this will answer some of your questions: https://medium.com/@knoldus/understanding-the-working-of-spark-driver-and-executor-4fec0e669399 .All the best,BS
- 2 kudos
- 5074 Views
- 12 replies
- 0 kudos
Multiple concurrent jobs using interactive cluster
Hi All,I have notebook in Databricks. This notebook is executed from azure datafactory pipeline having a databricks notebook activity with linkedservice connected to an interactive cluster.When multiple concurrent runs of this pipeline are created, I...
- 5074 Views
- 12 replies
- 0 kudos
- 0 kudos
Greetings @jigar191089 , I did some digging and here are some ideas to think about. This smells like a shared-state/import-path issue on an interactive cluster under concurrency. What likely happened Your notebook imports Python modules from /dbf...
- 0 kudos
- 165 Views
- 2 replies
- 1 kudos
Resolved! DLT | Communication lost with driver | Cluster was not reachable for 120 seconds
Hey Community, I'm facing this error, It says that "com.databricks.pipelines.common.errors.deployment.DeploymentException: Communication lost with driver. Cluster 1030-205818-yu28ft9s was not reachable for 120 seconds" This issue occurred in producti...
- 165 Views
- 2 replies
- 1 kudos
- 1 kudos
This is actually a known intermittent issue in Databricks, particularly with streaming or Delta Live Tables (DLT) pipelines.This isn’t a logical failure in your code — it’s an infrastructure-level timeout between the Databricks control plane and the ...
- 1 kudos
- 106 Views
- 1 replies
- 0 kudos
Pull workspace url and workspace name using databricks-sdk / programaticaly in notebook
1. How could I pull workspace url (https://adb-XXXXX.XX.....net) 2. How could I get workspace name visible in top right corner.I know that easies solution is dbutils.notebook.entry_point.... browserHostName but unfortunetly it is not working in job c...
- 106 Views
- 1 replies
- 0 kudos
- 0 kudos
Can you give this a shot? Not sure if you've a hard requirement of using SDK. workspace_url = spark.conf.get('spark.databricks.workspaceUrl') Getting name is more tricky. You could potentially get it from tags if there is a tagging strategy in place...
- 0 kudos
- 154 Views
- 1 replies
- 0 kudos
Databricks asset bundles CI/CD design for github actions
We are wanting to use Databricks asset bundles and deploy code changes and tests using github actions. We have seen lots of content online, but nothing concrete on how this is done at scale. So I'm wondering, if we have many changes and therefore man...
- 154 Views
- 1 replies
- 0 kudos
- 0 kudos
Have you read about following approach before? Repository Structure Options 1. Monorepo with Multiple Bundles repo-root/ ├── .github/ │ └── workflows/ │ ├── bundle-ci.yml │ └── bundle-deploy.yml ├── bundles/ │ ├...
- 0 kudos
- 104 Views
- 1 replies
- 0 kudos
Data Masking
Hi all,I need some help on this masking problem. If you create a view with used masking function based on table.The user reading this view has to have read access to underlying table. So theoretically, he can access unmasked data in the table.I would...
- 104 Views
- 1 replies
- 0 kudos
- 0 kudos
Are you on Unity catalog? Databricks has a solution for this through Unity Catalog Column Masking (also called Dynamic Views or Column-Level Security). https://learn.microsoft.com/en-us/azure/databricks/data-governance/unity-catalog/filters-and-mask...
- 0 kudos
- 131 Views
- 1 replies
- 0 kudos
Looking for reference DABs bundle yaml and resources for Databricks app deployment (FastAPI redirect
Looking for example databricks.yml and bundle resources for deploying a FastAPI Databricks app using DABs from one environment to another. Deployment works but FastAPI redirects to localhost after deployment, though the homepage loads fine. Need refe...
- 131 Views
- 1 replies
- 0 kudos
- 0 kudos
This is a great place to start: https://apps-cookbook.dev/resources/ Happy to answer specifics as they come after you've reviewed that resource.
- 0 kudos
- 6001 Views
- 8 replies
- 7 kudos
Azure DevOps Repo - Invalid Git Credentials
I have a Repo in Databricks connected to Azure DevOps Repositories.The repo has been working fine for almost a month, until last week. Now when I try to open the Git settings in Databricks, I am getting "Invalid Git Credentials". Nothing has change...
- 6001 Views
- 8 replies
- 7 kudos
- 7 kudos
I had a similar problem. I could fix following these steps:in the Azure Devops repository: User Settings -> Personal access tokens -> + New tokenin Databricks: Settings -> User -> Linked accounts -> Azure Devops (Personal access token)You could also...
- 7 kudos
| User | Count |
|---|---|
| 1619 | |
| 790 | |
| 485 | |
| 349 | |
| 287 |
