- 1649 Views
- 5 replies
- 7 kudos
From Associate to Professional: My Learning Plan to ace all Databricks Data Engineer Certifications
In today’s data-driven world, the role of a data engineer is critical in designing and maintaining the infrastructure that allows for the efficient collection, storage, and analysis of large volumes of data. Databricks certifications holds significan...
- 1649 Views
- 5 replies
- 7 kudos
- 7 kudos
@ms_ccg You are correct. I got that error too. Seems like Databricks has removed some of these. I would suggest you to search for those separately via Databricks Academy or external resources. Let me know if you need any help.
- 7 kudos
- 884 Views
- 1 replies
- 1 kudos
Spot databricks VMs - eviction rates
Before using Spot machines in #databricks, it's a good idea to check their eviction rates in your region. Azure Resource Graph Explorer and that simple query will help. SpotResources | where type =~ 'microsoft.compute/skuspotevictionrate/location' ...
- 884 Views
- 1 replies
- 1 kudos
- 1 kudos
Thank you for sharing this @Hubert-Dudek
- 1 kudos
- 1268 Views
- 0 replies
- 1 kudos
Installing CrowdStrike Falcon Sensor on Databricks Workers
Greetings,Does anyone here have experience deploying the CrowdStrike Falcon sensor on Databricks worker instances? For context, the cluster is deployed in AWS and we use a Databricks Ubuntu 20.04 AMI. Databricks allows adding a bootstrap/startup scri...
- 1268 Views
- 0 replies
- 1 kudos
- 1981 Views
- 0 replies
- 1 kudos
Post: Lakehouse Federation - Databricks
Lakehouse Federation - Databricks In the world of data, innovation is constant. And the most recent revolution comes with Lakehouse Federation, a fusion between data lakes and data warehouses, taking data manipulation to a new level. This advancement...
- 1981 Views
- 0 replies
- 1 kudos
- 721 Views
- 0 replies
- 0 kudos
Feedback request for Gradient, a tool to help optimize and monitor jobs automatically
Hi Everyone,We built Gradient, a tool to automatically optimize and monitor Databricks jobs to hit your business objectives of cost or runtime.Gradient works by applying a reinforcement ML model to automatically learn and custom tune your jobs cluste...
- 721 Views
- 0 replies
- 0 kudos
- 614 Views
- 0 replies
- 0 kudos
Understand why your jobs' performances are changing over time
Hi Folks -We released a new metrics view for databricks jobs in Gradient, which helps track and plot the metrics below over time to help engineers understand what's going on with their jobs over time.Job cost (DBU + Cloud fees)Job RuntimeNumber of co...
- 614 Views
- 0 replies
- 0 kudos
- 2312 Views
- 1 replies
- 1 kudos
Jonathan Frankel at Sigma talk
Hi @Sujitha Just to follow up on your suggestion to share my takeaways from Jonathan Frankel's talk at Sigma in NYC. The key ideas I came away with is:Building in-house custom models is more than just possible, there's advantages to itThere's danger...
- 2312 Views
- 1 replies
- 1 kudos
- 1 kudos
@Danny_Lee This is super insightful! Really appreciate your time to share your key takeaways with us.
- 1 kudos
- 1213 Views
- 0 replies
- 1 kudos
Databricks AI Security Framework
Today Databricks announced the release of the Databricks AI Security Framework (LinkedIn Post)You can download the paper (PDF) from blog post. Anyone else download this and have thoughts? My first thought is its a great start and has an excellent G...
- 1213 Views
- 0 replies
- 1 kudos
- 1149 Views
- 0 replies
- 0 kudos
GCP - Initial External Location to GCP Bucket is wrong
When creating a new Workspace in GCP the default GCP External Location is wrong.Its easily fixed by Catalog (on the left) > External Data (on the bottom) > External Locations > choose the connection and edit the URL by deleting the second BucketId af...
- 1149 Views
- 0 replies
- 0 kudos
- 1482 Views
- 0 replies
- 0 kudos
Predictive optimization log
After you enable predictive optimization, it is good to look at the system table and see what is going on with your tables #databricks
- 1482 Views
- 0 replies
- 0 kudos
- 5860 Views
- 3 replies
- 2 kudos
Materials to pass Databricks Data Engineering Associate Exam
Hi Guys, I have passed it already some time ago, but just recently have summarized all the materials which helped me to do it. Pay special attention to GitHub repository, which contains many great exercises prepared by Databricks teamhttps://youtu.be...
- 5860 Views
- 3 replies
- 2 kudos
- 1302 Views
- 0 replies
- 0 kudos
Feature article: Leveraging Generative AI with Apache Spark: Transforming Data Engineering
I created this article in Linkedlin to allow both this community and Apache Spark user community to have access to it.It is particularly useful for data engineers who want to have a basic understanding of what Generative AI with Spark can do.Leverag...
- 1302 Views
- 0 replies
- 0 kudos
- 3186 Views
- 1 replies
- 3 kudos
DBR 15.0 beta
databricks runtime 15 is out there!Some breaking changes. More info here https://docs.databricks.com/en/release-notes/runtime/15.0.html
- 3186 Views
- 1 replies
- 3 kudos
- 3 kudos
Thanks for sharing this information @Hubert-Dudek!!!
- 3 kudos
- 2625 Views
- 1 replies
- 1 kudos
Notebook IDE
This is an excellent step for #databricks notebooks. Integrated debugger and CLI in notebook terminal is a big step towards a fully functional cloud IDE.
- 2625 Views
- 1 replies
- 1 kudos
- 1 kudos
Thank you for sharing this @Hubert-Dudek!!!
- 1 kudos
- 3744 Views
- 2 replies
- 0 kudos
Build a machine learning model to detect fraudulent transactions using PySpark's MLlib library
IntroductionFinancial fraud is a significant concern for businesses and consumers alike. I have written about this concern a few times in Linkedlin articles. Machine learning offers powerful tools to combat this issue by automatically identifying sus...
- 3744 Views
- 2 replies
- 0 kudos
- 0 kudos
Looking to build a machine learning model for detecting fraudulent transactions using PySpark’s MLlib. Generate synthetic transaction data. Provides a dataset for model training without using sensitive real-world data. Enables the creation of diverse...
- 0 kudos
- 1398 Views
- 1 replies
- 2 kudos
is it possible to have a class level separation in databricks or implement a design pattern in datab
if you have thought about making your code inside databricks and notebooks more reusable and organized and you have thought about implementing a design pattern or class level separation in databricks the answer is yes, I am going to tell you the deta...
- 1398 Views
- 1 replies
- 2 kudos
- 2 kudos
tnx! I have spent quite some time on figuring out what the best way is. Your approach is certainly a valid one.Myself I prefer to package reused classes in a jar (we mainly code in scala). Works fine too.
- 2 kudos
