As data volume and complexity increase, engineers are left figuring out how to manage, monitor and maintain fragile pipelines while also handling fragmented tools. The Big Book of Data Engineering equips you with cutting-edge methods for building pip...
- 379 Views
- 0 replies
- 4 kudos
Wednesday
How to Register & Prepare If you're interested in advancing your skills with Databricks through a Specialist Session, here's a clear guide on how to register and what free courses you can take to prepare effectively. How to Begin Your Learning Path S...
- 2090 Views
- 0 replies
- 7 kudos
10-02-2025
This week was charged – from insightful discussions to Hackathon brilliance, our members showed off their creativity, collaboration, and unmatched Databricks energy! Let’s dive into this week’s highlights Community Stars of the Week Big cheers to e...
- 253 Views
- 3 replies
- 5 kudos
Friday
Hello Community, Let me start off with a quick question: Have you ever... Migrated your workloads from on-prem Spark to Databricks and encountered a bug and thought, “I wish I could repro this locally to debug the issue without burning cluster hours...
- 568 Views
- 8 replies
- 12 kudos
Thursday
Community Activity
- 7971 Views
- 16 replies
- 14 kudos
Free Certification voucher
looking for free certification voucher
- 7971 Views
- 16 replies
- 14 kudos
Latest Reply
- 14 kudos
Hi, can you please provide me free voucher for azure data bricks fundamental
- 14 kudos
- 17 Views
- 0 replies
- 0 kudos
POC on spark 4.x
I need to do some POC with spark 3.5.7 and 4.x and need some local setup with some sample Kafka source. The POC would read data from Kafka via streaming job and write to delta table and I would like to do this on spark-4.x ..Do you know of any quick ...
- 17 Views
- 0 replies
- 0 kudos
- 56 Views
- 4 replies
- 1 kudos
Resolved! Got an empty query file when cloning Query file.
Hello Community,In our AWS Databricks environment, we’ve encountered some behavior we don’t understand while performing the following operation.When we clone a query file that already has content, a new file is created with the same name and “(clone)...
- 56 Views
- 4 replies
- 1 kudos
Latest Reply
- 1 kudos
@kyeongmin_baek - There is no aut-save or CMD+S for Query as it get saved only once it is attached to a cluster AND 'SAVE' icon is used. However, it still stays in cache as unsaved in that 'query' window but cloning or other file operations may lose ...
- 1 kudos
- 43 Views
- 2 replies
- 0 kudos
Error while creating databricks custom app
I am trying to create a simple databricks custom app but I am getting Error: Could not import 'app'. error.app.yaml fileenv: - name: FLASK_APP value: '/Workspace/Users/sam@xxx.com/databricks_apps/hello-world_2025_11_13-16_19/Gaap_commentry/app'comm...
- 43 Views
- 2 replies
- 0 kudos
Latest Reply
- 0 kudos
Thanks, I have modified the yaml file but still getting Error: Could not import 'app' errorenv: - name: FLASK_APP value: '/Workspace/Users/xxx@zzz.com/databricks_apps/hello-world_2025_11_13-16_19/Gaap_commentry'command: [ "flask", "--app", "...
- 0 kudos
- 8 Views
- 0 replies
- 0 kudos
How to change the display name for a Service Principal
Hi,I'm trying to change the displayName for some EntraID managed service principals.I've tried using the scim API with a PATCH request, I get a HTTP 200 response, but the displayName remains the same.I tried the same with Databricks managed service ...
- 8 Views
- 0 replies
- 0 kudos
- 25 Views
- 1 replies
- 0 kudos
Usage Dashboard from Cody Austin Davis displays only DBUs or overall cost including the VM uptime?
Hello,I have been looking at the new Dashboard created by @CodyA ( great job! ) and I was wondering if the cost displaying only provides visibility to only the databricks mark-up on each job ( i.e. $ DBUs ) or to the overall cost including cloud prov...
- 25 Views
- 1 replies
- 0 kudos
Latest Reply
- 0 kudos
As the dashboard uses System Tables (system.billing.usage) to show spend across Jobs, SQL, and Notebooks, I dont beleive it includes cloud provider VM costs.
- 0 kudos
- 24 Views
- 4 replies
- 1 kudos
Want to see logs for lineage view run events
Hi All,I need your help, as I am running jobs it is getting successful, when I click on job and there we can find lineage > View run events option when click on it. I see below steps.Job Started: The job is triggered.Waiting for Cluster: The job wait...
- 24 Views
- 4 replies
- 1 kudos
Latest Reply
- 1 kudos
Hi Team/Member,As I am running jobs it is getting successful, when I click on job and there we can find lineage > View run events option when click on it. We find below steps and also added screenshot of it. I want screenshot stages logs, where i wil...
- 1 kudos
- 74 Views
- 5 replies
- 0 kudos
Accessing permanent file system from Databricks App
I have a Python-based FastAPI application with a React frontend, which I would like to deploy to Databricks apps. I'm able to do it, but having trouble with file system access. In addition to database access, my application relies on some file operat...
- 74 Views
- 5 replies
- 0 kudos
Latest Reply
- 0 kudos
Related thread, but would like to confirm:https://community.databricks.com/t5/administration-architecture/how-to-access-unitycatalog-s-volume-inside-databricks-app/td-p/121716
- 0 kudos
- 14 Views
- 1 replies
- 0 kudos
Access to lab
Hi All,I am new to databricks and even i am trying to access lab "vocareum" will this bee even possible as i am using https://partner-academy.databricks.com/or it there any possibility to access lab for the course -- Platform Administrator Learning P...
- 14 Views
- 1 replies
- 0 kudos
Latest Reply
- 0 kudos
Unfortunately No. Those labs are accessbile in the Instructor-led trainings only at least that I have had experience.
- 0 kudos
- 4251 Views
- 7 replies
- 6 kudos
Resolved! How to get Lab access provided by Vocareum
Hi TeamI have enrolled in the AI/BI for Data Analyst course. Could you please advise how to get the access for LAB to follow the demo instructions in the courseThanks
- 4251 Views
- 7 replies
- 6 kudos
Latest Reply
- 6 kudos
Hi Advika, I tried to get to the Lab suscription, but the weg page is not working. Could you please give me the correct link to access it?Thank you very much.
- 6 kudos
- 69 Views
- 4 replies
- 4 kudos
Resolved! Autoloader Managed File events
Hi all,We are in the process of migrating from directory listing to managed file events in Azure Databricks. Our data is stored in an Azure Data Lake container with the following folder structure:To enable file events in Unity Catalog (UC), I created...
- 69 Views
- 4 replies
- 4 kudos
Latest Reply
- 4 kudos
Recommended approach to continue your existing pattern:Keep the External Location enabled for file events at the high-level path (/Landing).Run a separate Structured Streaming job for each table, specifying the full sub-path in the .load() function (...
- 4 kudos
- 55 Views
- 3 replies
- 3 kudos
how to know which join type was used (broadcast, shuffle hash or sort merge join) for a query?
What is the best way to know what kind of join was used for a SQL query between broadcast, shuffle hash and sort merge? How can the spark UI or the query plan be interpreted?
- 55 Views
- 3 replies
- 3 kudos
Latest Reply
- 3 kudos
@smoortema , Spark performance tuning is one of the hardest topics to teach or learn, and it’s even tougher to do justice to in a forum thread. That said, I’m really glad to see you asking the question. Tuning is challenging precisely because there a...
- 3 kudos
- 1321 Views
- 7 replies
- 4 kudos
create an external connection to oracle
Hi! I've been trying to create an external connection to oracle but getting the following error message "Detailed error message: ORA-00604: error occurred at recursive SQL level 1 ORA-01882: timezone region not found" I searched online and found some...
- 1321 Views
- 7 replies
- 4 kudos
Latest Reply
- 4 kudos
hey @austinoyoung ,I don't have an Oracle database to be able to test this for you, but I believe you can get around this error by following the steps laid out in here:https://stackoverflow.com/questions/9156379/ora-01882-timezone-region-not-foundIn ...
- 4 kudos
- 24 Views
- 2 replies
- 1 kudos
Resolved! Unable to navigate/login to Databricks Account Console
Hi,I have deployed Azure Databricks using email id (say xx@gmail.com) and able to lauch a workspace.When I try to access account console, it throws below errorSelected user account does not exist in tenant 'Microsoft Services' and cannot access the ...
- 24 Views
- 2 replies
- 1 kudos
Latest Reply
- 1 kudos
And old link but still relevant - https://github.com/cloudboxacademy/azure_databricks_course/blob/main/known-issues/unable-to-login-to-azure-databricks-account-console.md
- 1 kudos
- 25186 Views
- 27 replies
- 26 kudos
Resolved! Unable to login to Azure Databricks Account Console
I have a personal Azure pay-as-you-go subscription in which I have the 'Global Administrator' role. I am also the databricks account administrator.Until two weeks ago, I was able to access the databricks account console without any issues, but I am f...
- 25186 Views
- 27 replies
- 26 kudos
Latest Reply
- 26 kudos
Thanks Dustin.This solves my issue too,but I want to know WHY this happened?I used an email id (say,xx@gmail.com) to login to Azure,using the same id/user I have deployed databricks and able to launch the workspace.But not account console.what's so s...
- 26 kudos
Announcements
| User | Count |
|---|---|
| 2123 | |
| 1183 | |
| 995 | |
| 948 | |
| 858 |
Latest from our Blog
Upgrade Your HMS and AWS Glue Tables: How-To Guide for Migrating Foreign Tables to Unity Catalog
Intro Unlock Unity Catalog governance and performance by upgrading Hive Metastore (HMS) and AWS Glue foreign tables to Managed Tables using the new Upgrade Foreign Table workflow. Managed Tables provi...
From Lag to Agility: Reinventing Freshworks’ Data Ingestion Architecture
As a global software-as-a-service (SaaS) company specializing in providing intuitive, AI-powered business solutions designed to enhance customer and employee experiences, Freshworks depends on real-ti...
From Hackathon to Impact: Building a Data-Driven Forecasting App on Databricks
IntroductionAfter Databricks' Data + AI Summit, an electrifying hackathon lit up the London HQ. Teams from across industries came together to build bold, tangible solutions using DBX’s latest features...
How to Deploy Metric Views with DABs
Introduction Unity Catalog Metric Views provide the ability to build a semantic layer in Databricks, using YAML and SQL. This shifts the semantic layer left into your catalog, moving it closer to your...
