Governing the Full Data Lifecycle: From Ingestion to Insight with Unity Catalog Date: Saturday, December 13 Time: 12:00 PM – 3:00 PM Location: Bengaluru, Karnataka (Register to see address) Registration: Approval required Register Here! About Thi...
- 67 Views
- 1 replies
- 2 kudos
12 hours ago
BrickTalk: Vibe-Coding Databricks Apps in Replit Discover how to vibe-code Databricks Apps in Replit and accelerate your development workflow. Follow along as we demonstrate techniques for going from concept to demo in record time. Join us for a han...
- 915 Views
- 4 replies
- 4 kudos
2 weeks ago
Our Champion program has always celebrated the customers who go above and beyond to engage, help others, and uplift the Community. Recently, we have seen remarkable participation from Bricksters as well—and their impact deserves recognition too. Begi...
- 628 Views
- 6 replies
- 12 kudos
2 weeks ago
As data volume and complexity increase, engineers are left figuring out how to manage, monitor and maintain fragile pipelines while also handling fragmented tools. The Big Book of Data Engineering equips you with cutting-edge methods for building pip...
- 962 Views
- 5 replies
- 9 kudos
3 weeks ago
How to Register & Prepare If you're interested in advancing your skills with Databricks through a Specialist Session, here's a clear guide on how to register and what free courses you can take to prepare effectively. How to Begin Your Learning Path S...
- 2598 Views
- 2 replies
- 8 kudos
10-02-2025
Hello Community, Let me start off with a quick question: Have you ever... Migrated your workloads from on-prem Spark to Databricks and encountered a bug and thought, “I wish I could repro this locally to debug the issue without burning cluster hours...
- 1700 Views
- 12 replies
- 17 kudos
3 weeks ago
A warm and vibrant week in the Databricks Community! Even with the Thanksgiving celebrations in full swing, the Community kept the spark alive – sharing insights, solving tricky questions, and dropping some solid technical gems.Here’s your weekly rou...
- 257 Views
- 5 replies
- 6 kudos
Friday
Community Activity
- 614 Views
- 4 replies
- 0 kudos
Databricks Asset Bundle Deployment Fails in GitHub Actions with Federated Identity Credentials
I am using a service principal with workspace admin access to deploy Databricks asset bundles. The deployment works successfully via Jenkins using the same credentials and commands. However, when attempting the deployment through GitHub Actions, I en...
- 614 Views
- 4 replies
- 0 kudos
Latest Reply
- 0 kudos
I recently ran into a similar issue attempting to deploy with the Terraform Databricks provider:provider "databricks" {host = <host>azure_workspace_resource_id = <workspace_resource_id>azure_client_id = <azure_client_id>azure_client_secret = <azure_c...
- 0 kudos
- 47 Views
- 5 replies
- 3 kudos
Resolved! Databricks Free-Edition - Hive_Metastore
Hi,Today I went to access the Databricks Free_Edition catalog: Legacy / Hive_Metastore and it wasn't there.Is it no longer available in this version or have they changed it?Thanks,Fabio
- 47 Views
- 5 replies
- 3 kudos
- 3348 Views
- 4 replies
- 2 kudos
Find value in any column in a table
Hi,I'm not sure if this is a possible scenario, but is there, by any chance a way to query all the columns of a table for searching a value? Explanation: I want to search for a specific value in all the columns of a databricks table. I don't know whi...
- 3348 Views
- 4 replies
- 2 kudos
- 15 Views
- 1 replies
- 0 kudos
Databricks academy error setup - Free Edition with Serverless Compute
Databricks is passing from the Community Edition to the Free Edition, which I am currently using.When executing the Includes/Classroom-setup notebooks the following exception is raised: [CONFIG_NOT_AVAILABLE] Configuration dbacademy.deprecation.loggi...
- 15 Views
- 1 replies
- 0 kudos
Latest Reply
- 0 kudos
@iFoxz17 The CONFIG_NOT_AVAILABLE error (SQLSTATE: 42K0I) occurs because spark.conf.get(key) in Databricks Free Edition throws an exception for missing configuration keys like dbacademy.deprecation.logging, instead of returning the default value as e...
- 0 kudos
- 2287 Views
- 8 replies
- 0 kudos
DELTA LIVE TABLE -Parallel processing
how can we process multiple tables within a delta live table pipeline parallelly as table names as parameters.
- 2287 Views
- 8 replies
- 0 kudos
Latest Reply
- 0 kudos
Maybe better approach will be to use dlt meta https://databrickslabs.github.io/dlt-meta/index.html
- 0 kudos
- 27 Views
- 3 replies
- 0 kudos
Create a table from external location volume using terraform
Hi everyone,I’m trying to create a table in Unity Catalog from a folder that lives inside a Volume, but I can’t get Terraform to create it successfully.Below is the simplified/anonymous version of my Terraform setup.I have:* A Unity Catalog* A Schema...
- 27 Views
- 3 replies
- 0 kudos
Latest Reply
- 0 kudos
It will not work because it will detect that the locations overlap. I tried it in some experiments, like displaying delta_log in volumes, but it didn't work. In fact, it is secure this way as the Delta files in volume would bypass Unity Catalog ACL f...
- 0 kudos
- 11 Views
- 1 replies
- 0 kudos
Genie - Value dictionary
When I add the tables to the genie space, it automatically turns on the value dictionary for first 120 string fields, Is there a way to disable them by default and add only to the needed fields later?I am working in curating responses for around 15 t...
- 11 Views
- 1 replies
- 0 kudos
Latest Reply
- 0 kudos
"Value sampling is enabled by default for all Genie spaces." The only solution that I see is to build json and use as a serialized_space template https://docs.databricks.com/api/azure/workspace/genie/getspace
- 0 kudos
- 12 Views
- 0 replies
- 0 kudos
How to add a line break for Data labels in Visualization Editor
How can I add a break line between data labels in Visualization Editor?Not seeing anything in the documentation that helpsI would like the data labels in the example below to show as:trade count: 223Price: 84.71875
- 12 Views
- 0 replies
- 0 kudos
- 1700 Views
- 12 replies
- 22 kudos
⭐ Setup Spark with Hadoop Anywhere : A DBR aligned local Spark+HDFS+Hive stack on Docker⭐
Hello Community, Let me start off with a quick question: Have you ever... Migrated your workloads from on-prem Spark to Databricks and encountered a bug and thought, “I wish I could repro this locally to debug the issue without burning cluster hours...
- 1700 Views
- 12 replies
- 22 kudos
Latest Reply
- 22 kudos
Thank you for sharing this. I have tried this in the past and gave up at certain point. It is helpful
- 22 kudos
- 49 Views
- 6 replies
- 2 kudos
UC volumes not useable in Apps?
I have to install some custom library in a Python Databricks App. According to the documentation this should be possible through UC volumes:https://docs.databricks.com/aws/en/dev-tools/databricks-apps/dependencies#install-wheel-files-from-unity-catal...
- 49 Views
- 6 replies
- 2 kudos
- 98 Views
- 5 replies
- 0 kudos
Create external table using iceberg not working
I'm trying to create a external iceberg table on top of files I have in s3 created using a spark job that runs outside of databricks using Hadoop catalog.Create External table table_name using iceberg location 's3://myicebergbucket/iceberg_table_path...
- 98 Views
- 5 replies
- 0 kudos
Latest Reply
- 0 kudos
Databricks doesn’t support the LOCATION clause for Iceberg in Unity Catalog; Iceberg tables must be created as managed tables without a path, or read via a foreign catalog. Trying USING iceberg LOCATION 's3://...' triggers “Managed Iceberg tables do ...
- 0 kudos
- 42 Views
- 2 replies
- 0 kudos
How to Optimize Data Pipeline Development on Databricks for Large-Scale Workloads?
Hi everyone,I’m working on building and optimizing data pipelines in Databricks, especially for large-scale workloads, and I want to learn from others who have hands-on experience with performance tuning, architecture decisions, and best practices.I’...
- 42 Views
- 2 replies
- 0 kudos
Latest Reply
- 0 kudos
Yes, +1 to @szymon_dybczak This is the best doc to start with on the optimization part - https://www.databricks.com/discover/pages/optimize-data-workloads-guide
- 0 kudos
- 7320 Views
- 9 replies
- 10 kudos
Clean up _delta_log files
Hello experts. We are trying to clarify how to clean up the large amount of files that are being accumulated in the _delta_log folder (json, crc and checkpoint files). We went through the related posts in the forum and followed the below:SET spark.da...
- 7320 Views
- 9 replies
- 10 kudos
Latest Reply
- 10 kudos
Delta Lake does automatically clean up _delta_log files (JSON, CHECKPOINT, CRC), but only when two conditions are met: The retention durations are respectedBy default: delta.logRetentionDuration = 30 days delta.deletedFileRetentionDuration = 7 day...
- 10 kudos
- 13 Views
- 0 replies
- 0 kudos
Guest Lecture by Databricks expert Sai Yang on Big Data and AI
We are delighted to share more about our most recent Databricks guest lecture delivered by Sai Yang on big data and AI at Bonn-Rhein-Sieg University of Applied Sciences.The aim of the event was to link theoretically learned concepts with concrete pra...
- 13 Views
- 0 replies
- 0 kudos
- 14996 Views
- 22 replies
- 4 kudos
spotify API get token - raw_input was called, but this frontend does not support input requests.
hello everyone, I'm trying use spotify's api to analyse my music data, but i'm receiving a error during authentication, specifically when I try get the token, above my code.Is it a databricks bug?pip install spotipyfrom spotipy.oauth2 import SpotifyO...
- 14996 Views
- 22 replies
- 4 kudos
Announcements
| User | Count |
|---|---|
| 2134 | |
| 1200 | |
| 1021 | |
| 948 | |
| 863 |
Latest from our Blog
Driving Revenue with Data: How Vehicle Service Notes Unlock Upsell Opportunities
The Hidden Story in Every Service Visit It’s a busy Tuesday morning at a dealership. A customer pulls in for what should be a simple oil change. The technician performs the inspection, then notices so...
Revolutionizing GenAI Application Management: MLflow 3 Prompt Registry in Databricks
In enterprise GenAI deployments, prompts are the critical interface between users and AI models—yet most organizations manage them like scattered text files. This creates bottlenecks that prevent GenA...
Upgrade Your HMS and AWS Glue Tables: How-To Guide for Migrating Foreign Tables to Unity Catalog
Intro Unlock Unity Catalog governance and performance by upgrading Hive Metastore (HMS) and AWS Glue foreign tables to Managed Tables using the new Upgrade Foreign Table workflow. Managed Tables provi...
From Lag to Agility: Reinventing Freshworks’ Data Ingestion Architecture
As a global software-as-a-service (SaaS) company specializing in providing intuitive, AI-powered business solutions designed to enhance customer and employee experiences, Freshworks depends on real-ti...
