- 1614 Views
- 5 replies
- 7 kudos
From Associate to Professional: My Learning Plan to ace all Databricks Data Engineer Certifications
In today’s data-driven world, the role of a data engineer is critical in designing and maintaining the infrastructure that allows for the efficient collection, storage, and analysis of large volumes of data. Databricks certifications holds significan...
- 1614 Views
- 5 replies
- 7 kudos
- 7 kudos
@ms_ccg You are correct. I got that error too. Seems like Databricks has removed some of these. I would suggest you to search for those separately via Databricks Academy or external resources. Let me know if you need any help.
- 7 kudos
- 1966 Views
- 0 replies
- 1 kudos
Post: Lakehouse Federation - Databricks
Lakehouse Federation - Databricks In the world of data, innovation is constant. And the most recent revolution comes with Lakehouse Federation, a fusion between data lakes and data warehouses, taking data manipulation to a new level. This advancement...
- 1966 Views
- 0 replies
- 1 kudos
- 714 Views
- 0 replies
- 0 kudos
Feedback request for Gradient, a tool to help optimize and monitor jobs automatically
Hi Everyone,We built Gradient, a tool to automatically optimize and monitor Databricks jobs to hit your business objectives of cost or runtime.Gradient works by applying a reinforcement ML model to automatically learn and custom tune your jobs cluste...
- 714 Views
- 0 replies
- 0 kudos
- 608 Views
- 0 replies
- 0 kudos
Understand why your jobs' performances are changing over time
Hi Folks -We released a new metrics view for databricks jobs in Gradient, which helps track and plot the metrics below over time to help engineers understand what's going on with their jobs over time.Job cost (DBU + Cloud fees)Job RuntimeNumber of co...
- 608 Views
- 0 replies
- 0 kudos
- 2292 Views
- 1 replies
- 1 kudos
Jonathan Frankel at Sigma talk
Hi @Sujitha Just to follow up on your suggestion to share my takeaways from Jonathan Frankel's talk at Sigma in NYC. The key ideas I came away with is:Building in-house custom models is more than just possible, there's advantages to itThere's danger...
- 2292 Views
- 1 replies
- 1 kudos
- 1 kudos
@Danny_Lee This is super insightful! Really appreciate your time to share your key takeaways with us.
- 1 kudos
- 1203 Views
- 0 replies
- 1 kudos
Databricks AI Security Framework
Today Databricks announced the release of the Databricks AI Security Framework (LinkedIn Post)You can download the paper (PDF) from blog post. Anyone else download this and have thoughts? My first thought is its a great start and has an excellent G...
- 1203 Views
- 0 replies
- 1 kudos
- 1148 Views
- 0 replies
- 0 kudos
GCP - Initial External Location to GCP Bucket is wrong
When creating a new Workspace in GCP the default GCP External Location is wrong.Its easily fixed by Catalog (on the left) > External Data (on the bottom) > External Locations > choose the connection and edit the URL by deleting the second BucketId af...
- 1148 Views
- 0 replies
- 0 kudos
- 1478 Views
- 0 replies
- 0 kudos
Predictive optimization log
After you enable predictive optimization, it is good to look at the system table and see what is going on with your tables #databricks
- 1478 Views
- 0 replies
- 0 kudos
- 5722 Views
- 3 replies
- 2 kudos
Materials to pass Databricks Data Engineering Associate Exam
Hi Guys, I have passed it already some time ago, but just recently have summarized all the materials which helped me to do it. Pay special attention to GitHub repository, which contains many great exercises prepared by Databricks teamhttps://youtu.be...
- 5722 Views
- 3 replies
- 2 kudos
- 1295 Views
- 0 replies
- 0 kudos
Feature article: Leveraging Generative AI with Apache Spark: Transforming Data Engineering
I created this article in Linkedlin to allow both this community and Apache Spark user community to have access to it.It is particularly useful for data engineers who want to have a basic understanding of what Generative AI with Spark can do.Leverag...
- 1295 Views
- 0 replies
- 0 kudos
- 3176 Views
- 1 replies
- 3 kudos
DBR 15.0 beta
databricks runtime 15 is out there!Some breaking changes. More info here https://docs.databricks.com/en/release-notes/runtime/15.0.html
- 3176 Views
- 1 replies
- 3 kudos
- 3 kudos
Thanks for sharing this information @Hubert-Dudek!!!
- 3 kudos
- 2620 Views
- 1 replies
- 1 kudos
Notebook IDE
This is an excellent step for #databricks notebooks. Integrated debugger and CLI in notebook terminal is a big step towards a fully functional cloud IDE.
- 2620 Views
- 1 replies
- 1 kudos
- 1 kudos
Thank you for sharing this @Hubert-Dudek!!!
- 1 kudos
- 3689 Views
- 2 replies
- 0 kudos
Build a machine learning model to detect fraudulent transactions using PySpark's MLlib library
IntroductionFinancial fraud is a significant concern for businesses and consumers alike. I have written about this concern a few times in Linkedlin articles. Machine learning offers powerful tools to combat this issue by automatically identifying sus...
- 3689 Views
- 2 replies
- 0 kudos
- 0 kudos
Looking to build a machine learning model for detecting fraudulent transactions using PySpark’s MLlib. Generate synthetic transaction data. Provides a dataset for model training without using sensitive real-world data. Enables the creation of diverse...
- 0 kudos
- 1395 Views
- 1 replies
- 2 kudos
is it possible to have a class level separation in databricks or implement a design pattern in datab
if you have thought about making your code inside databricks and notebooks more reusable and organized and you have thought about implementing a design pattern or class level separation in databricks the answer is yes, I am going to tell you the deta...
- 1395 Views
- 1 replies
- 2 kudos
- 2 kudos
tnx! I have spent quite some time on figuring out what the best way is. Your approach is certainly a valid one.Myself I prefer to package reused classes in a jar (we mainly code in scala). Works fine too.
- 2 kudos
- 4374 Views
- 1 replies
- 1 kudos
Building Event-Driven Real-Time Data Processor with Spark Structured Streaming and API Integration
I recently saw an article from Databricks titled "Scalable Spark Structured Streaming for REST API Destinations". A great article focusing on continuous Spark Structured Streaming (SSS). About a year old. I then decided, given customer demands to wo...
- 4374 Views
- 1 replies
- 1 kudos
- 1 kudos
I recently saw an article from Databricks titled "Scalable Spark Structured Streaming for REST API Destinations". A great article focusing on continuous Spark Structured Streaming (SSS). About a year old. I then decided, given customer demands to wo...
- 1 kudos
- 1568 Views
- 0 replies
- 0 kudos
stored procedures
The plan for stored procedures in databricks spark has been announced in a few places. How can stored procedures look in Spark SQL?
- 1568 Views
- 0 replies
- 0 kudos
