Get Started Resources
Explore essential resources to kickstart your journey with Databricks. Access tutorials, guides, and...
0 Posts
Activity in Resources
- 30 Views
- 0 replies
- 0 kudos
Jumpstart Your Data Journey with Databricks Get Started Days!
Are you ready to dive into the world of data engineering and analytics? Join us for Databricks Get Started Days, a half-day virtual event designed to accelerate your learning and equip you with essential Databricks skills! Here’s what makes Get Start...
- 30 Views
- 0 replies
- 0 kudos
- 3453 Views
- 4 replies
- 12 kudos
Performance Tuning using Query Profile
Introduction In the fast-paced world of big data, optimizing performance is critical for maintaining efficiency and reducing costs. Databricks SQL (DBSQL) Warehouse is a robust feature of the Databricks platform that enables data analysts, data engin...
- 3453 Views
- 4 replies
- 12 kudos
Latest Reply
- 12 kudos
Very detailed and informative. Thanks @rakhidarshi for sharing!
- 12 kudos
- 33 Views
- 0 replies
- 1 kudos
The Hidden Pitfalls of Snowflake to Databricks Migrations
Everyone's rushing their Snowflake to Databricks migration, and they're setting themselves up for failure.After leading multiple enterprise migrations to Databricks last quarter, here's what shocked me: The technical lift isn't the hard part. It's th...
- 33 Views
- 0 replies
- 1 kudos
- 8184 Views
- 3 replies
- 3 kudos
Materials to pass Databricks Data Engineering Associate Exam
Hi Guys, I have passed it already some time ago, but just recently have summarized all the materials which helped me to do it. Pay special attention to GitHub repository, which contains many great exercises prepared by Databricks teamhttps://youtu.be...
- 8184 Views
- 3 replies
- 3 kudos
Latest Reply
- 3 kudos
I almost gave up after failing Databricks-Certified-Data-Engineer-Associate once, but then I found Passexamhub. Their realistic practice tests helped me understand my weak areas and improve. Passed on my second attempt—such a relief!
- 3 kudos
- 1210 Views
- 0 replies
- 6 kudos
Introducing SAP Databricks
Today we are announcing a deep partnership with SAP which we think can be game changing for our industry. In short, it is the marriage between the most important business data for enterprises globally (SAP data) and the best data platform in the mark...
- 1210 Views
- 0 replies
- 6 kudos
- 2814 Views
- 5 replies
- 28 kudos
Best practices for safe data experimentation with Databricks
In today’s data-centric world, experimentation is essential for developers and data scientists to create cutting-edge models, test hypotheses, and build robust data pipelines. However, giving these teams access to production data raises serious conce...
- 2814 Views
- 5 replies
- 28 kudos
Latest Reply
- 28 kudos
Great post!I went after more reading while reading each topics and I would like to add a few things here1. AnonymizationI wouldn't use uuid() like this.Using a hashing function would be better to ensure consistency across multiple runs. F.sha2(F.con...
- 28 kudos
- 55906 Views
- 18 replies
- 32 kudos
Metadata-Driven ETL Framework in Databricks (Part-1)
In modern data-driven enterprises, data flows like lifeblood through complex systems and repositories to drive decision-making and innovation. Each dataset, whether structured or unstructured, holds the potential to unlock insights and drive innovati...
- 55906 Views
- 18 replies
- 32 kudos
Latest Reply
- 32 kudos
It is a great article, I am excited for the next parts.I am not sure about having these metadata tables in the lakehouse. It forces us to build a data pipeline for the metadata table. Isn't it better to just use a transactional database like mongo or...
- 32 kudos
- 3010 Views
- 2 replies
- 5 kudos
Private Cross-Cloud Delta Sharing with Serverless Compute
As organizations increasingly adopt multi-cloud strategies to leverage the unique strengths of various cloud platforms, they face the dual challenge of maintaining robust security while enabling efficient data sharing. Balancing accessibility with p...
- 3010 Views
- 2 replies
- 5 kudos
Latest Reply
- 5 kudos
Thank you for your feedback @Mantsama4!! Our solution is designed to tackle both B2B and Line of Business sharing by combining robust security with flexible, region-specific deployment. For B2B scenarios, we ensure that external partners access data ...
- 5 kudos
- 881 Views
- 0 replies
- 2 kudos
How to Read Unity Catalog Tables from Trino via Open APIs
Databricks Unity Catalog (UC) is the industry’s only unified and open governance solution for data and AI, built into the Databricks Data Intelligence Platform. Unity Catalog provides a single source of truth for your organization’s data and AI asset...
- 881 Views
- 0 replies
- 2 kudos
- 343 Views
- 0 replies
- 1 kudos
Ray Monitoring Made Easy: Prometheus & Grafana with Ray on Databricks Clusters
Intro Ray is rapidly becoming the standard for logic-parallel computing, enabling many Databricks customers to accelerate a wide range of Python workloads. Since its general availability on Databricks in early 2024, Ray on Databricks has opened up ne...
- 343 Views
- 0 replies
- 1 kudos
- 1956 Views
- 1 replies
- 4 kudos
Deep Dive - Streaming Deduplication
In this article we will cover in depth about streaming deduplication using watermarking with dropDuplicates and dropDuplicatesWithinWatermark, how they are different. This blog expects you to have a good understanding on how watermarking works in Spa...
- 1956 Views
- 1 replies
- 4 kudos
Latest Reply
- 4 kudos
Hi @MuraliTalluri,Thank you for such a detailed article.I am following dropDuplicatesWithinWatermark with the same steps as yours. The only difference is that I am using autoloader and reading CSV files as the source and writing data to a Delta table...
- 4 kudos
- 1068 Views
- 3 replies
- 4 kudos
Unlocking Cost and Performance Insights in Databricks with Custom Dashboards
As organizations continue to scale their data infrastructure, efficient resource utilization, cost control, and operational transparency are paramount for success. With the growing adoption of Databricks, monitoring and optimizing compute usage and d...
- 1068 Views
- 3 replies
- 4 kudos
Latest Reply
- 4 kudos
Thank you Mohana for sharing the detail, really appreciate it.
- 4 kudos
- 821 Views
- 1 replies
- 1 kudos
📊 Simplifying CDC with Databricks Delta Live Tables & Snapshots 📊
In the world of data integration, synchronizing external relational databases (like Oracle, MySQL) with the Databricks platform can be complex, especially when Change Data Feed (CDF) streams aren’t available. Using snapshots is a powerful way to mana...
- 821 Views
- 1 replies
- 1 kudos
Latest Reply
- 1 kudos
Hi AjayCan apply changes into snapshot handle re-processing of an older snapshot? UseCase:- Source has delivered data on day T, T1 and T2. - Consumers realise there is an error on the day T data, and make a correction in the source. The source redel...
- 1 kudos
- 94842 Views
- 6 replies
- 30 kudos
Top 10 query performance tuning tips for Databricks Serverless SQL
Databricks Serverless SQL (DBSQL) is the latest offering from Databricks to build data warehouses on the Lakehouse. It incorporates all the Lakehouse features like open format, unified analytics, and collaborative platforms across the different data ...
- 94842 Views
- 6 replies
- 30 kudos
Latest Reply
- 30 kudos
This is a great solution! The post provides an in-depth, structured approach to optimizing Databricks SQL Serverless, highlighting key tips such as resource optimization, query performance improvements, and the best practices for data types and cachi...
- 30 kudos
- 541 Views
- 1 replies
- 2 kudos
Consideration Before Migrating Hive Tables to Unity Catalog
Databricks recommends four methods to migrate Hive tables to Unity Catalog, each with its pros and cons. The choice of method depends on specific requirements.SYNC: A SQL command that migrates schema or tables to Unity Catalog external tables. Howeve...
- 541 Views
- 1 replies
- 2 kudos
Latest Reply
- 2 kudos
This is a great solution! The post effectively outlines the methods for migrating Hive tables to Unity Catalog while emphasizing the importance of not just performing a simple migration but transforming the data architecture into something more robus...
- 2 kudos
