Join us for the return of the Databricks Learning Festival (Virtual)! Mark your calendars from 15 January - 31 January 2025! Upskill today across data engineering, data analysis, machine learning, and generative AI. Join the thousands who have el...
- 103479 Views
- 240 replies
- 67 kudos
11-26-2024
Your opinion matters! Take a few minutes to complete our Customer Experience Survey to help us improve the Databricks Community. Your input is crucial in shaping the future of our community and ensuring it meets your needs. Take the Survey Now Why p...
- 874 Views
- 0 replies
- 0 kudos
2 weeks ago
We’re thrilled to share that Databricks has once again been recognized as a Leader in the 2024 Gartner® Magic Quadrant™ for Cloud Database Management Systems. This acknowledgement underscores our commitment to innovation and our leadership in the dat...
- 1744 Views
- 0 replies
- 3 kudos
3 weeks ago
We are thrilled to announce that DatabricksTV, our growing video hub, has hit a major milestone: 100 videos and counting! What is DatabricksTV?DatabricksTV is a community-driven video hub designed to help data practitioners maximize the Databricks e...
- 1724 Views
- 1 replies
- 4 kudos
12-11-2024
The Meta Llama 3.3 multilingual large language model (LLM) is a pretrained and instruction tuned generative model in 70B (text in/text out). The Llama 3.3 instruction tuned text only model is optimized for multilingual dialogue use cases and outperfo...
- 2498 Views
- 0 replies
- 3 kudos
12-11-2024
Community Activity
- 9 Views
- 1 replies
- 0 kudos
Upcoming St. Louis User Group Meetings?
Are there any upcoming St. Louis User Group meetings coming up? Can anyone tell me who is coordinating them?
- 9 Views
- 1 replies
- 0 kudos
Latest Reply
- 0 kudos
Hi @MarkWasson ! Yes, the St. Louis User Group is coming up soon! Here's the event page where you can signup.
- 0 kudos
- 6 Views
- 0 replies
- 0 kudos
PowerBI reports migration from Synapse to Databricks sql
Hello Techie,Do someone please share detailed design consideration when migrating 250 powerBI reports from Azure Synapse, to Databricks (DB SQL).How to plan for cutover and strategy for PowerBI. what are the consideration to be done , is there any da...
- 6 Views
- 0 replies
- 0 kudos
- 30 Views
- 3 replies
- 0 kudos
Cluster creation issue
Hello,I just started using Databricks community version for learning purposes.I have been trying to create a cluster but the first time it failed asking me to retry or contact the support, and now it's just running forever.What could be the problem?
- 30 Views
- 3 replies
- 0 kudos
Latest Reply
- 0 kudos
Can you please check the cluster logs, in the Cluster page, there is a logs section, that should help throw some light. Cluster Running Forever - it also could be resource availability issue or config, but ideally it should fail, again logs, can help...
- 0 kudos
- 2 Views
- 0 replies
- 0 kudos
What's Your Databricks Aha! Moment?🌟
What's Your Databricks Aha! Moment? Whether you’re new to Databricks or a seasoned pro, there’s always that one moment when everything just clicked. Maybe it was when you nailed a tricky notebook, optimized a query, solved a big data challenge, or ...
- 2 Views
- 0 replies
- 0 kudos
- 2 Views
- 0 replies
- 0 kudos
What's Your Databricks Aha! Moment?🌟
What's Your Databricks Aha! Moment? Whether you’re new to Databricks or a seasoned pro, there’s always that one moment when everything just clicked. Maybe it was when you nailed a tricky notebook, optimized a query, solved a big data challenge, or ...
- 2 Views
- 0 replies
- 0 kudos
- 3 Views
- 0 replies
- 0 kudos
What's Your Databricks Aha! Moment?🌟
What's Your Databricks Aha! Moment? Whether you’re new to Databricks or a seasoned pro, there’s always that one moment when everything just clicked. Maybe it was when you nailed a tricky notebook, optimized a query, solved a big data challenge, or ...
- 3 Views
- 0 replies
- 0 kudos
- 17 Views
- 2 replies
- 0 kudos
How to connect from Serverless Plane to On-Prem SQL Server
so has anybody tried connecting Databricks Serverless in the Serverless plane to on-prem SQL server. we can connect databricks normal cluster with federated queries with External Data connections to on-prem SQL serverwe can connect Serverless to Azu...
- 17 Views
- 2 replies
- 0 kudos
Latest Reply
- 0 kudos
oh thanks for investigating. for the serverless --> azure storage account , I created an NCC and created Private Endpoint rules for the storage accounts, and associated the NCC with the workspace. I'm guessing this is not the Private Link config yo...
- 0 kudos
- 9 Views
- 0 replies
- 0 kudos
Notifications have file information but dataframe is empty using autoloader file notification mode
Using DBR 13.3, i'm ingesting data from 1 adls storage account using autoloader with file notification mode enabled. and writing to container in another adls storage account. This is an older code which is using foreachbatch sink to process the data ...
- 9 Views
- 0 replies
- 0 kudos
- 34 Views
- 1 replies
- 0 kudos
R model deployment
unable to serve R model on databricks even tried with pyfunc
- 34 Views
- 1 replies
- 0 kudos
Latest Reply
- 0 kudos
Are you getting any specific error message?
- 0 kudos
- 14 Views
- 1 replies
- 0 kudos
Question about Data Management for Supply-Demand Allocation
I have a scenario where I am trying to parallelize supply - demand allotment between sellers and buyers with many to many links. I am unsure of whether I can parallelize the calculation using PySpark operations. I have two columns to keep track of in...
- 14 Views
- 1 replies
- 0 kudos
Latest Reply
- 0 kudos
Parallelizing supply-demand allotment in PySpark can be challenging due to the need for sequential updates to supply and demand values across rows. However, it is possible to achieve this using PySpark operations, though it may require a different ap...
- 0 kudos
- 44 Views
- 1 replies
- 0 kudos
Resolved! DNSResolve Error while establishing JDBC connection to Azure Databricks
I am using the Databricks JDBC driver (https://databricks.com/spark/jdbc-drivers-download) to connect to Azure Databricks through a VPN.I am connecting through a SAAS low-code platform, Appian, so I don't have access to any more logs. We have set up ...
- 44 Views
- 1 replies
- 0 kudos
Latest Reply
- 0 kudos
It seems that the DNS is not being able to resolve the domain name of your workspace, from the browser with the VPN connection are you able to access to it?
- 0 kudos
- 145 Views
- 3 replies
- 0 kudos
How to remove legacy hive metasore under catalog section ?
Respected Concern,We are currently using unity catalog in our workspace but we still see legacy hive_metastore under catalog section. Is there any way we can remove it. Because the issue we are facing is our cluster still somehow tries to connect wit...
- 145 Views
- 3 replies
- 0 kudos
Latest Reply
- 0 kudos
Error we encounter is below:Errors:Caused by: org.datanucleus.exceptions.NucleusException: Attempt to invoke the "HikariCP" plugin to create a ConnectionPool gave an error: Failed to initialize pool: Could not connect to address=(host=xyz.region.rds....
- 0 kudos
- 162 Views
- 6 replies
- 0 kudos
Resolved! DLT Pipeline Event Logs
There seems to be a issue now with our DLT pipeline event logs. I am not sure if this is a recent bug or not (but they were ok in Dec). But the issue is in dev, qc and prod and we only have a couple days of history logs now visible in the UI.From wha...
- 162 Views
- 6 replies
- 0 kudos
- 193 Views
- 5 replies
- 0 kudos
Resolved! Updating Workspace Cluster
Hello,My organization is experiencing difficulties updating our Google Kubernetes Engine (GKE) cluster.We've reviewed the official GKE documentation for automated cluster updates, but it appears to primarily focus on AWS integrations. We haven't foun...
- 193 Views
- 5 replies
- 0 kudos
- 539 Views
- 7 replies
- 0 kudos
Handling Audit Columns and SCD Type 1 in Databricks DLT Pipeline with Unity Catalog: Circular Depend
I am working on a Delta Live Tables (DLT) pipeline with Unity Catalog, where we are reading data from Azure Data Lake Storage (ADLS) and creating a table in the silver layer with Slowly Changing Dimensions (SCD) Type 1 enabled. In addition, we are ad...
- 539 Views
- 7 replies
- 0 kudos
Latest Reply
- 0 kudos
When a SCD Type 1 is processed using the APPLY CHANGES INTO (or APPLY FROM SNAPSHOT) APIs there are not columns like the "__CreatedDateTime" and "__UpdatedDateTime" automatically created in the structure and maintained through the APPLY. That is unl...
- 0 kudos
Announcements
Latest from our Blog
Mastering RAG Chatbot Security: ACL and Metadata Filtering with Mosaic AI Vector Search
Imagine you’re running a company with multiple departments, like Finance, Legal, and HR. Each department has its own sensitive data—financial reports, legal contracts, and employee records—that need t...
Deep Dive - Streaming Deduplication
In this article we will cover in depth about streaming deduplication using watermarking with dropDuplicates and dropDuplicatesWithinWatermark, how they are different. This blog expects you to have a g...
Data Engineering SQL Holiday Specials
December is the most celebrated time of year in the Data Engineering calendar as we embrace the important holiday: change freeze season. As we come back to the office to start our new projects, I wan...
Federated Data Catalog Ownership: Balance Governance and Autonomy
The Context: Centers of Excellence and Data StrategyThree Models for Data Catalog OwnershipOption 1: Decentralized OwnershipOption 2: Centralized Creation by Platform TeamOption 3: Hybrid Model with D...
