- 13404 Views
- 2 replies
- 7 kudos
Materials to pass Databricks Data Engineering Associate Exam
Hi Guys, I have passed it already some time ago, but just recently have summarized all the materials which helped me to do it. Pay special attention to GitHub repository, which contains many great exercises prepared by Databricks teamhttps://youtu.be...
- 13404 Views
- 2 replies
- 7 kudos
- 1845 Views
- 0 replies
- 0 kudos
Feature article: Leveraging Generative AI with Apache Spark: Transforming Data Engineering
I created this article in Linkedlin to allow both this community and Apache Spark user community to have access to it.It is particularly useful for data engineers who want to have a basic understanding of what Generative AI with Spark can do.Leverag...
- 1845 Views
- 0 replies
- 0 kudos
- 4003 Views
- 1 replies
- 3 kudos
DBR 15.0 beta
databricks runtime 15 is out there!Some breaking changes. More info here https://docs.databricks.com/en/release-notes/runtime/15.0.html
- 4003 Views
- 1 replies
- 3 kudos
- 3 kudos
Thanks for sharing this information @Hubert-Dudek!!!
- 3 kudos
- 3446 Views
- 1 replies
- 1 kudos
Notebook IDE
This is an excellent step for #databricks notebooks. Integrated debugger and CLI in notebook terminal is a big step towards a fully functional cloud IDE.
- 3446 Views
- 1 replies
- 1 kudos
- 1 kudos
Thank you for sharing this @Hubert-Dudek!!!
- 1 kudos
- 6253 Views
- 2 replies
- 0 kudos
Build a machine learning model to detect fraudulent transactions using PySpark's MLlib library
IntroductionFinancial fraud is a significant concern for businesses and consumers alike. I have written about this concern a few times in Linkedlin articles. Machine learning offers powerful tools to combat this issue by automatically identifying sus...
- 6253 Views
- 2 replies
- 0 kudos
- 0 kudos
Looking to build a machine learning model for detecting fraudulent transactions using PySpark’s MLlib. Generate synthetic transaction data. Provides a dataset for model training without using sensitive real-world data. Enables the creation of diverse...
- 0 kudos
- 1952 Views
- 1 replies
- 2 kudos
is it possible to have a class level separation in databricks or implement a design pattern in datab
if you have thought about making your code inside databricks and notebooks more reusable and organized and you have thought about implementing a design pattern or class level separation in databricks the answer is yes, I am going to tell you the deta...
- 1952 Views
- 1 replies
- 2 kudos
- 2 kudos
tnx! I have spent quite some time on figuring out what the best way is. Your approach is certainly a valid one.Myself I prefer to package reused classes in a jar (we mainly code in scala). Works fine too.
- 2 kudos
- 5954 Views
- 1 replies
- 1 kudos
Building Event-Driven Real-Time Data Processor with Spark Structured Streaming and API Integration
I recently saw an article from Databricks titled "Scalable Spark Structured Streaming for REST API Destinations". A great article focusing on continuous Spark Structured Streaming (SSS). About a year old. I then decided, given customer demands to wo...
- 5954 Views
- 1 replies
- 1 kudos
- 2176 Views
- 0 replies
- 0 kudos
stored procedures
The plan for stored procedures in databricks spark has been announced in a few places. How can stored procedures look in Spark SQL?
- 2176 Views
- 0 replies
- 0 kudos
- 952 Views
- 0 replies
- 0 kudos
Redesigned Move File & Clone File Experiences
Hi everyone! We are redesigning the Move File and Clone File experiences. We want to make it as seamless as possible to organize your files, and would love your feedback on the designs! Move File: Move Option 1 Move Option 2: Clone File: Cl...
- 952 Views
- 0 replies
- 0 kudos
- 2248 Views
- 1 replies
- 2 kudos
liquid partitioning
Based on my experience with data partitioning, it often diminishes performance rather than enhancing it. There are exceptions, like when handling tables over 1TB, or when EVERY single query utilizes partition in the WHERE clause - for instance, a Pow...
- 2248 Views
- 1 replies
- 2 kudos
- 2 kudos
Thank you for sharing this @Hubert-Dudek !!
- 2 kudos
- 3157 Views
- 0 replies
- 2 kudos
Your updated resource guide to the Databricks Data Intelligence Platform
Want to increase your Databricks knowledge? Look no further!Here’s a guide filled with key resources you’ll need while working on the Databricks Data Inteligence Platform. Bookmark these pages for future reference, or apply these learnings during the...
- 3157 Views
- 0 replies
- 2 kudos
- 4554 Views
- 0 replies
- 0 kudos
✨New in Notebooks: AI-powered Databricks Assistant, improved visualizations, web terminal and more!
We are excited to share some of the latest updates in Databricks Notebooks. From AI-powered Databricks Assistant that automates code development to new charts with better performance, these features help you build faster. See the latest features live...
- 4554 Views
- 0 replies
- 0 kudos
- 11452 Views
- 1 replies
- 1 kudos
Simplify complex workflows with modular jobs
Thousands of Databricks customers use Databricks Workflows every day to orchestrate business-critical workloads on the Databricks Lakehouse Platform. A great way to simplify those critical workloads is through modular orchestration. This is now possi...
- 11452 Views
- 1 replies
- 1 kudos
- 1 kudos
Hello, is the "if/else condition" task type available for testing?
- 1 kudos
- 9029 Views
- 0 replies
- 0 kudos
Set up AI-driven optimizations in Databricks SQL
With Predictive I/O for reads (GA) and updates (Public Preview), Databricks SQL can now analyze historical read and write patterns to intelligently build indexes and optimize DELETE, MERGE, and UPDATE operations. What is Predictive I/O? Predictive I/...
- 9029 Views
- 0 replies
- 0 kudos
- 10780 Views
- 0 replies
- 0 kudos
Share views and schemas with your team
Delta Sharing is a great way to securely share data across different Unity Catalog metastores in your own Databricks account. This now includes using Delta Sharing to share views and schemas directly with other Databricks recipients from within the D...
- 10780 Views
- 0 replies
- 0 kudos
