Date & Time:5 March (Wednesday), 11 AM – 3 PM SGT What You'll Learn:In this training session, you will learn how to self-serve business insights from Databricks Data Intelligence Platform using AI/BI. After a tour of the fundamental components of t...
- 133 Views
- 0 replies
- 0 kudos
Sunday
The Databricks Developer Relations team is excited to share that we are hitting the road and bringing a series of community meetups to you called DevConnect! These events will be hosted around the globe and tailored specifically for Data engineers. T...
- 94 Views
- 2 replies
- 0 kudos
yesterday
Are you ready to dive into the world of data engineering and analytics? Join us for Databricks Get Started Days, a half-day virtual event designed to accelerate your learning and equip you with essential Databricks skills! Here’s what makes Get Start...
- 37 Views
- 0 replies
- 0 kudos
9 hours ago
Today we are announcing a deep partnership with SAP which we think can be game changing for our industry. In short, it is the marriage between the most important business data for enterprises globally (SAP data) and the best data platform in the mark...
- 1215 Views
- 0 replies
- 6 kudos
Thursday
We’re thrilled to announce the General Availability (GA) of Databricks Clean Rooms on AWS and Azure, a significant step forward in enabling secure, privacy-centric data collaboration. Following the Public Preview launch in August, we've engaged close...
- 793 Views
- 0 replies
- 1 kudos
2 weeks ago
Community Activity
- 4 Views
- 0 replies
- 0 kudos
using concurrent.futures for parallelization
Hi, trying to copy a table with billions of rows from an enterprise data source into my databricks table. To do this, I need to use a homegrown library which handles auth etc, runs the query and return a dataframe. I am partitioning the table using...
- 4 Views
- 0 replies
- 0 kudos
- 170 Views
- 3 replies
- 2 kudos
Unable to create Unity Catalog Metastore
Hi,I deployed a Unity Catalog metastore some time back (screenshot below).But when I am trying to create a new one, the create metastore option is disabled. Hovering on the button shows the message: "You must have an active subscription to create a M...
- 170 Views
- 3 replies
- 2 kudos
Latest Reply
- 2 kudos
Are you using Azure Databricks ?? have you recently migrated your classic Global Administration role to RBAC global admin in azure we had the same issue we fix it by creating metastore from CLI databricks metastores create {Name} --region {regi...
- 2 kudos
- 43 Views
- 1 replies
- 1 kudos
Resolved! Timeout settings for Postgresql external catalog connection?
Is there any way to configure timeouts for external catalog connections? We are getting some timeouts with complex queries accessing a pgsql database through the catalog. We tried configuring the connection and we got this error │ Error: cannot upda...
- 43 Views
- 1 replies
- 1 kudos
Latest Reply
- 1 kudos
Hello @ErikApption, there is no direct support for a connectTimeout option in the connection settings through Unity Catalog as of now. You might need to explore these alternative timeout configurations or consider adjusting your database handling to ...
- 1 kudos
- 35 Views
- 1 replies
- 0 kudos
Writing a small pyspark dataframe to a table is taking a very long time
My experience with Databricks pyspark up to this point has always been to execute a SQL query against existing Databricks tables, then write the resulting pyspark dataframe into a new table. For the first time, I am now getting data via an API which ...
- 35 Views
- 1 replies
- 0 kudos
Latest Reply
- 0 kudos
Can you share the code? You need to remember that spark uses lazy evaluation so it can give you impression that code works fast and saving works slowly because a code is executed when you it hit an action.
- 0 kudos
- 45 Views
- 1 replies
- 0 kudos
How to create Service Principal and access APIs like clusters list without adding to admin group
I have created a Databricks Managed Service Principal and trying to access the APIs like clusters list, job lists pipelines but without adding it to admin group I am getting empty list in response. There are other ways to get clusters by adding polic...
- 45 Views
- 1 replies
- 0 kudos
Latest Reply
- 0 kudos
Hi @akshay716, You can assign specific permissions directly to the service principal without granting it broader admin access
- 0 kudos
- 23 Views
- 0 replies
- 0 kudos
Dataiku connector limitation
Hello,I'm trying to read data from Unity Catalog and insert it into an Oracle Database using an "On Premise" Dataiku.It works well for a small dataset ~600Kb/~150 000 rows.[14:51:20] [INFO] [dku.datasets.sql] - Read 2000 records from DB [14:51:20] [I...
- 23 Views
- 0 replies
- 0 kudos
- 31 Views
- 0 replies
- 0 kudos
can view allow_expectations_col in unit catalog
I am developing a dlt that manages expectations and it works correctly.but I need to see the columns__DROP_EXPECTATIONS_COL__MEETS_DROP_EXPECTATIONS__ALLOW_EXPECTATIONS_COLin the unified catalog, I can see them in the delta table that the dlt generat...
- 31 Views
- 0 replies
- 0 kudos
- 30 Views
- 0 replies
- 0 kudos
Cant read/write tables with shared cluster
Hi!I have a pipeline that i cant execute sucessfully in a shared cluster. Basically i read a query from multiple sources on my databricks instance, including streaming tables (thats the reason i have to use a shared cluster).But when comes to the par...
- 30 Views
- 0 replies
- 0 kudos
- 281 Views
- 3 replies
- 0 kudos
Cannot create a workspace on GCP
Hi,I have been using Databricks for a couple of months and been spinning up workspaces with Terraform. The other day we decided to end our POC and move on to a MVP. This meant cleaning up all workspaces and GCP. after the cleanup was done I wanted to...
- 281 Views
- 3 replies
- 0 kudos
Latest Reply
- 0 kudos
Did you try from Marketplace? You may get there more detailed error.
- 0 kudos
- 211 Views
- 6 replies
- 2 kudos
Efficacy of PySpark in Databricks
Hi all,- migrating from an on-premise Oracle -Currently on Oracle, I have a "library" of let's say 300 tables to load, sequentially, based on views (some tables being fed potentially by several views, therefore the number of underlying views is highe...
- 211 Views
- 6 replies
- 2 kudos
Latest Reply
- 2 kudos
The common scenario for data processing in Oracle is based on PL/SQL and cursors. In the case of PySpark we don't have such concept as cursors and iteration on data frames can lead to poor performance.I migrated Oracle to Databricks and I learned tha...
- 2 kudos
- 30 Views
- 1 replies
- 0 kudos
When Sign in databricks partner-academy i can not see the courses
Dear partner academy team, I am writing to report an issue I am experiencing when trying to access the partner academy courses. Despite using my credentials, I am unable to view any of the courses. Could you please look into this and assist me in res...
- 30 Views
- 1 replies
- 0 kudos
Latest Reply
- 0 kudos
Hello @Sergio_Linares! Please file a ticket with the Databricks support team to get assistance with this issue.
- 0 kudos
- 149 Views
- 3 replies
- 4 kudos
Turn Off Auto-reveal of Navigation Sidebar
I work with the navigation sidebar closed and use the stacked hamburgers symbol in the upper left to reveal it when I want. Now, if you mouse over the left edge of the browser window too slowly it will auto-reveal the navigation sidebar. I do not wan...
- 149 Views
- 3 replies
- 4 kudos
Latest Reply
- 4 kudos
This behavior is unbelievably annoying and has been driving me to distraction since it started. I would love for there to be a way to keep the side menu pinned closed.
- 4 kudos
- 94 Views
- 4 replies
- 0 kudos
Access for delta lake with serverless
I have an issue when trying to use the command display(dbutils.fs.ls("abfss://test@test.dfs.core.windows.net")). When I execute the command on my personal cluster, it works, and I can see the files. Before that, I set the following configurations:spa...
- 94 Views
- 4 replies
- 0 kudos
Latest Reply
- 0 kudos
Can your serverless compute access any storage in that storage account? Something else to check is if your NCC is configured correctly: Configure private connectivity from serverless compute - Azure Databricks | Microsoft Learn. However, if your se...
- 0 kudos
- 37 Views
- 0 replies
- 0 kudos
Jumpstart Your Data Journey with Databricks Get Started Days!
Are you ready to dive into the world of data engineering and analytics? Join us for Databricks Get Started Days, a half-day virtual event designed to accelerate your learning and equip you with essential Databricks skills! Here’s what makes Get Start...
- 37 Views
- 0 replies
- 0 kudos
- 20 Views
- 0 replies
- 0 kudos
Join Us for Data Intelligence Day in Mountain View!
Thursday, March 6, 2025 8:00 AM – 5:00 PM PT The Ameswell Hotel, 800 Moffett Blvd, Mountain View, CA 94043 Secure your spot today! Register Here! Join us for a half-day deep dive into the future of data and AI. Learn how to: Eliminate data silos ...
- 20 Views
- 0 replies
- 0 kudos
Announcements
| User | Count |
|---|---|
| 106 | |
| 43 | |
| 24 | |
| 14 | |
| 13 |
Latest from our Blog
How to Read Unity Catalog Tables from Trino via Open APIs
Databricks Unity Catalog (UC) is the industry’s only unified and open governance solution for data and AI, built into the Databricks Data Intelligence Platform. Unity Catalog provides a single source ...
Ray Monitoring Made Easy: Prometheus & Grafana with Ray on Databricks Clusters
Intro Ray is rapidly becoming the standard for logic-parallel computing, enabling many Databricks customers to accelerate a wide range of Python workloads. Since its general availability on Databricks...
[CUSTOMER BLOG] Efficient Distributed Energy Load Forecasting with Databricks at EDP E-REDES
Traditionally, power grids have been sized with large safety margins to consider low-probability events. Thus, there can be limits imposed on generation even when the low-probability events do not occ...
Improved Duplicate Match Handling in Delta MERGE Operations
In DBR 16.1+, we’ve improved functionality of MERGE operations where multiple rows of the source dataset match the same row of the target Delta table, but only one row matches the WHEN MATCHED conditi...
