cancel
Showing results for 
Search instead for 
Did you mean: 
cancel
Showing results for 
Search instead for 
Did you mean: 
Intelligent Data Warehousing: AI/BI for Self-service Analytics

 Date & Time:5 March (Wednesday), 11 AM – 3 PM SGT  What You'll Learn:In this training session, you will learn how to self-serve business insights from Databricks Data Intelligence Platform using AI/BI. After a tour of the fundamental components of t...

  • 133 Views
  • 0 replies
  • 0 kudos
Sunday
Databricks DevConnect: Global Community Meetups for Data Engineers

The Databricks Developer Relations team is excited to share that we are hitting the road and bringing a series of community meetups to you called DevConnect! These events will be hosted around the globe and tailored specifically for Data engineers. T...

  • 94 Views
  • 2 replies
  • 0 kudos
yesterday
Jumpstart Your Data Journey with Databricks Get Started Days!

Are you ready to dive into the world of data engineering and analytics? Join us for Databricks Get Started Days, a half-day virtual event designed to accelerate your learning and equip you with essential Databricks skills! Here’s what makes Get Start...

  • 37 Views
  • 0 replies
  • 0 kudos
9 hours ago
Introducing SAP Databricks

Today we are announcing a deep partnership with SAP which we think can be game changing for our industry. In short, it is the marriage between the most important business data for enterprises globally (SAP data) and the best data platform in the mark...

  • 1215 Views
  • 0 replies
  • 6 kudos
Thursday
Databricks Clean Rooms: Now Generally Available on AWS and Azure

We’re thrilled to announce the General Availability (GA) of Databricks Clean Rooms on AWS and Azure, a significant step forward in enabling secure, privacy-centric data collaboration. Following the Public Preview launch in August, we've engaged close...

  • 793 Views
  • 0 replies
  • 1 kudos
2 weeks ago

Community Activity

ironv
by > Visitor
  • 4 Views
  • 0 replies
  • 0 kudos

using concurrent.futures for parallelization

Hi, trying to copy a table with billions of rows from an enterprise data source into my databricks table.  To do this, I need to use a homegrown library which handles auth etc, runs the query and return a dataframe.  I am partitioning the table using...

  • 4 Views
  • 0 replies
  • 0 kudos
mscbatra
by > New Contributor II
  • 170 Views
  • 3 replies
  • 2 kudos

Unable to create Unity Catalog Metastore

Hi,I deployed a Unity Catalog metastore some time back (screenshot below).But when I am trying to create a new one, the create metastore option is disabled. Hovering on the button shows the message: "You must have an active subscription to create a M...

mscbatra_0-1739311719720.png
  • 170 Views
  • 3 replies
  • 2 kudos
Latest Reply
anujsen18
New Contributor II
  • 2 kudos

Are you using Azure Databricks ?? have you  recently migrated your  classic  Global Administration role  to RBAC global admin in azure we had the same issue we fix it by creating  metastore from CLI databricks metastores create {Name} --region {regi...

  • 2 kudos
2 More Replies
ErikApption
by > New Contributor II
  • 43 Views
  • 1 replies
  • 1 kudos

Resolved! Timeout settings for Postgresql external catalog connection?

Is there any way to configure timeouts for external catalog connections? We are getting some timeouts with complex queries accessing a pgsql database through the catalog. We tried configuring the connection and we got this error  │ Error: cannot upda...

  • 43 Views
  • 1 replies
  • 1 kudos
Latest Reply
Alberto_Umana
Databricks Employee
  • 1 kudos

Hello @ErikApption, there is no direct support for a connectTimeout option in the connection settings through Unity Catalog as of now. You might need to explore these alternative timeout configurations or consider adjusting your database handling to ...

  • 1 kudos
philHarasz
by > New Contributor
  • 35 Views
  • 1 replies
  • 0 kudos

Writing a small pyspark dataframe to a table is taking a very long time

My experience with Databricks pyspark up to this point has always been to execute a SQL query against existing Databricks tables, then write the resulting pyspark dataframe into a new table. For the first time, I am now getting data via an API which ...

  • 35 Views
  • 1 replies
  • 0 kudos
Latest Reply
MariuszK
Contributor II
  • 0 kudos

Can you share the code? You need to remember that spark uses lazy evaluation so it can give you impression that code works fast and saving works slowly because a code is executed when you it hit an action.

  • 0 kudos
akshay716
by > Visitor
  • 45 Views
  • 1 replies
  • 0 kudos

How to create Service Principal and access APIs like clusters list without adding to admin group

I have created a Databricks Managed Service Principal and trying to access the APIs like clusters list, job lists pipelines but without adding it to admin group I am getting empty list in response. There are other ways to get clusters by adding polic...

  • 45 Views
  • 1 replies
  • 0 kudos
Latest Reply
Alberto_Umana
Databricks Employee
  • 0 kudos

Hi @akshay716, You can assign specific permissions directly to the service principal without granting it broader admin access

  • 0 kudos
MaximeGendre
by > New Contributor III
  • 23 Views
  • 0 replies
  • 0 kudos

Dataiku connector limitation

Hello,I'm trying to read data from Unity Catalog and insert it into an Oracle Database using an "On Premise" Dataiku.It works well for a small dataset ~600Kb/~150 000 rows.[14:51:20] [INFO] [dku.datasets.sql] - Read 2000 records from DB [14:51:20] [I...

MaximeGendre_0-1739993758870.png
  • 23 Views
  • 0 replies
  • 0 kudos
tebodelpino1234
by > Visitor
  • 31 Views
  • 0 replies
  • 0 kudos

can view allow_expectations_col in unit catalog

I am developing a dlt that manages expectations and it works correctly.but I need to see the columns__DROP_EXPECTATIONS_COL__MEETS_DROP_EXPECTATIONS__ALLOW_EXPECTATIONS_COLin the unified catalog, I can see them in the delta table that the dlt generat...

tebodelpino1234_0-1739993377613.png tebodelpino1234_1-1739993707990.png tebodelpino1234_2-1739993775604.png
  • 31 Views
  • 0 replies
  • 0 kudos
Ramonrcn
by > New Contributor III
  • 30 Views
  • 0 replies
  • 0 kudos

Cant read/write tables with shared cluster

Hi!I have a pipeline that i cant execute sucessfully in a shared cluster. Basically i read a query from multiple sources on my databricks instance, including streaming tables (thats the reason i have to use a shared cluster).But when comes to the par...

  • 30 Views
  • 0 replies
  • 0 kudos
Christian_j
by > New Contributor
  • 281 Views
  • 3 replies
  • 0 kudos

Cannot create a workspace on GCP

Hi,I have been using Databricks for a couple of months and been spinning up workspaces with Terraform. The other day we decided to end our POC and move on to a MVP. This meant cleaning up all workspaces and GCP. after the cleanup was done I wanted to...

  • 281 Views
  • 3 replies
  • 0 kudos
Latest Reply
MariuszK
Contributor II
  • 0 kudos

Did you try from Marketplace? You may get there more detailed error.

  • 0 kudos
2 More Replies
VCA50380
by > New Contributor III
  • 211 Views
  • 6 replies
  • 2 kudos

Efficacy of PySpark in Databricks

Hi all,- migrating from an on-premise Oracle -Currently on Oracle, I have a "library" of let's say 300 tables to load, sequentially, based on views (some tables being fed potentially by several views, therefore the number of underlying views is highe...

  • 211 Views
  • 6 replies
  • 2 kudos
Latest Reply
MariuszK
Contributor II
  • 2 kudos

The common scenario for data processing in Oracle is based on PL/SQL and cursors. In the case of PySpark we don't have such concept as cursors and iteration on data frames can lead to poor performance.I migrated Oracle to Databricks and I learned tha...

  • 2 kudos
5 More Replies
Sergio_Linares
by > Visitor
  • 30 Views
  • 1 replies
  • 0 kudos

When Sign in databricks partner-academy i can not see the courses

Dear partner academy team, I am writing to report an issue I am experiencing when trying to access the partner academy courses. Despite using my credentials, I am unable to view any of the courses. Could you please look into this and assist me in res...

  • 30 Views
  • 1 replies
  • 0 kudos
Latest Reply
Advika
Databricks Employee
  • 0 kudos

Hello @Sergio_Linares! Please file a ticket with the Databricks support team to get assistance with this issue.

  • 0 kudos
robbdunlap
by > New Contributor II
  • 149 Views
  • 3 replies
  • 4 kudos

Turn Off Auto-reveal of Navigation Sidebar

I work with the navigation sidebar closed and use the stacked hamburgers symbol in the upper left to reveal it when I want. Now, if you mouse over the left edge of the browser window too slowly it will auto-reveal the navigation sidebar. I do not wan...

  • 149 Views
  • 3 replies
  • 4 kudos
Latest Reply
LJacobsen
New Contributor III
  • 4 kudos

This behavior is unbelievably annoying and has been driving me to distraction since it started. I would love for there to be a way to keep the side menu pinned closed.

  • 4 kudos
2 More Replies
DataEnginerrOO1
by > Visitor
  • 94 Views
  • 4 replies
  • 0 kudos

Access for delta lake with serverless

I have an issue when trying to use the command display(dbutils.fs.ls("abfss://test@test.dfs.core.windows.net")). When I execute the command on my personal cluster, it works, and I can see the files. Before that, I set the following configurations:spa...

  • 94 Views
  • 4 replies
  • 0 kudos
Latest Reply
Rjdudley
Valued Contributor II
  • 0 kudos

Can your serverless compute access any storage in that storage account?  Something else to check is if your NCC is configured correctly: Configure private connectivity from serverless compute - Azure Databricks | Microsoft Learn.  However, if your se...

  • 0 kudos
3 More Replies
Sujitha
by Databricks Employee
  • 37 Views
  • 0 replies
  • 0 kudos

Jumpstart Your Data Journey with Databricks Get Started Days!

Are you ready to dive into the world of data engineering and analytics? Join us for Databricks Get Started Days, a half-day virtual event designed to accelerate your learning and equip you with essential Databricks skills! Here’s what makes Get Start...

Screenshot 2025-02-18 at 9.25.12 PM.png
  • 37 Views
  • 0 replies
  • 0 kudos
Rishabh_Tiwari
by Databricks Employee
  • 20 Views
  • 0 replies
  • 0 kudos

Join Us for Data Intelligence Day in Mountain View!

  Thursday, March 6, 2025 8:00 AM – 5:00 PM PT The Ameswell Hotel, 800 Moffett Blvd, Mountain View, CA 94043 Secure your spot today! Register Here! Join us for a half-day deep dive into the future of data and AI. Learn how to: Eliminate data silos ...

Rishabh_Tiwari_1-1739981598597.png Screenshot 2025-02-19 at 9.11.20 PM.png
  • 20 Views
  • 0 replies
  • 0 kudos

Connect with Databricks Users in Your Area

Join a Regional User Group to connect with local Databricks users. Events will be happening in your city, and you won’t want to miss the chance to attend and share knowledge.

If there isn’t a group near you, start one and help create a community that brings people together.

Request a New Group
Top Kudoed Authors
Read Databricks Data Intelligence Platform reviews on G2

Latest from our Blog

How to Read Unity Catalog Tables from Trino via Open APIs

Databricks Unity Catalog (UC) is the industry’s only unified and open governance solution for data and AI, built into the Databricks Data Intelligence Platform. Unity Catalog provides a single source ...

886Views 2kudos

Improved Duplicate Match Handling in Delta MERGE Operations

In DBR 16.1+, we’ve improved functionality of MERGE operations where multiple rows of the source dataset match the same row of the target Delta table, but only one row matches the WHEN MATCHED conditi...

586Views 3kudos