- 728 Views
- 1 replies
- 0 kudos
Spot databricks VMs - eviction rates
Before using Spot machines in #databricks, it's a good idea to check their eviction rates in your region. Azure Resource Graph Explorer and that simple query will help. SpotResources | where type =~ 'microsoft.compute/skuspotevictionrate/location' ...
- 728 Views
- 1 replies
- 0 kudos
- 0 kudos
Thank you for sharing this @Hubert-Dudek
- 0 kudos
- 972 Views
- 0 replies
- 0 kudos
Installing CrowdStrike Falcon Sensor on Databricks Workers
Greetings,Does anyone here have experience deploying the CrowdStrike Falcon sensor on Databricks worker instances? For context, the cluster is deployed in AWS and we use a Databricks Ubuntu 20.04 AMI. Databricks allows adding a bootstrap/startup scri...
- 972 Views
- 0 replies
- 0 kudos
- 1737 Views
- 0 replies
- 1 kudos
Post: Lakehouse Federation - Databricks
Lakehouse Federation - Databricks In the world of data, innovation is constant. And the most recent revolution comes with Lakehouse Federation, a fusion between data lakes and data warehouses, taking data manipulation to a new level. This advancement...
- 1737 Views
- 0 replies
- 1 kudos
- 1869 Views
- 1 replies
- 1 kudos
Jonathan Frankel at Sigma talk
Hi @Sujitha Just to follow up on your suggestion to share my takeaways from Jonathan Frankel's talk at Sigma in NYC. The key ideas I came away with is:Building in-house custom models is more than just possible, there's advantages to itThere's danger...
- 1869 Views
- 1 replies
- 1 kudos
- 1 kudos
@Danny_Lee This is super insightful! Really appreciate your time to share your key takeaways with us.
- 1 kudos
- 1001 Views
- 0 replies
- 1 kudos
Databricks AI Security Framework
Today Databricks announced the release of the Databricks AI Security Framework (LinkedIn Post)You can download the paper (PDF) from blog post. Anyone else download this and have thoughts? My first thought is its a great start and has an excellent G...
- 1001 Views
- 0 replies
- 1 kudos
- 1365 Views
- 1 replies
- 0 kudos
Predictive optimization log
After you enable predictive optimization, it is good to look at the system table and see what is going on with your tables #databricks
- 1365 Views
- 1 replies
- 0 kudos
- 0 kudos
Hi @Hubert-Dudek , Thank you for sharing your valuable insights on the Knowledge Sharing Hub. Your expertise is greatly appreciated and contributes significantly to our community's knowledge base.
- 0 kudos
- 1081 Views
- 0 replies
- 0 kudos
GCP - Initial External Location to GCP Bucket is wrong
When creating a new Workspace in GCP the default GCP External Location is wrong.Its easily fixed by Catalog (on the left) > External Data (on the bottom) > External Locations > choose the connection and edit the URL by deleting the second BucketId af...
- 1081 Views
- 0 replies
- 0 kudos
- 3483 Views
- 3 replies
- 2 kudos
Materials to pass Databricks Data Engineering Associate Exam
Hi Guys, I have passed it already some time ago, but just recently have summarized all the materials which helped me to do it. Pay special attention to GitHub repository, which contains many great exercises prepared by Databricks teamhttps://youtu.be...
- 3483 Views
- 3 replies
- 2 kudos
- 1152 Views
- 0 replies
- 0 kudos
Feature article: Leveraging Generative AI with Apache Spark: Transforming Data Engineering
I created this article in Linkedlin to allow both this community and Apache Spark user community to have access to it.It is particularly useful for data engineers who want to have a basic understanding of what Generative AI with Spark can do.Leverag...
- 1152 Views
- 0 replies
- 0 kudos
- 2949 Views
- 1 replies
- 3 kudos
DBR 15.0 beta
databricks runtime 15 is out there!Some breaking changes. More info here https://docs.databricks.com/en/release-notes/runtime/15.0.html
- 2949 Views
- 1 replies
- 3 kudos
- 3 kudos
Thanks for sharing this information @Hubert-Dudek!!!
- 3 kudos
- 2502 Views
- 1 replies
- 1 kudos
Notebook IDE
This is an excellent step for #databricks notebooks. Integrated debugger and CLI in notebook terminal is a big step towards a fully functional cloud IDE.
- 2502 Views
- 1 replies
- 1 kudos
- 1 kudos
Thank you for sharing this @Hubert-Dudek!!!
- 1 kudos
- 2954 Views
- 2 replies
- 0 kudos
Build a machine learning model to detect fraudulent transactions using PySpark's MLlib library
IntroductionFinancial fraud is a significant concern for businesses and consumers alike. I have written about this concern a few times in Linkedlin articles. Machine learning offers powerful tools to combat this issue by automatically identifying sus...
- 2954 Views
- 2 replies
- 0 kudos
- 0 kudos
Looking to build a machine learning model for detecting fraudulent transactions using PySpark’s MLlib. Generate synthetic transaction data. Provides a dataset for model training without using sensitive real-world data. Enables the creation of diverse...
- 0 kudos
- 1264 Views
- 1 replies
- 2 kudos
is it possible to have a class level separation in databricks or implement a design pattern in datab
if you have thought about making your code inside databricks and notebooks more reusable and organized and you have thought about implementing a design pattern or class level separation in databricks the answer is yes, I am going to tell you the deta...
- 1264 Views
- 1 replies
- 2 kudos
- 2 kudos
tnx! I have spent quite some time on figuring out what the best way is. Your approach is certainly a valid one.Myself I prefer to package reused classes in a jar (we mainly code in scala). Works fine too.
- 2 kudos
- 3966 Views
- 3 replies
- 2 kudos
Resolved! Building Event-Driven Real-Time Data Processor with Spark Structured Streaming and API Integration
I recently saw an article from Databricks titled "Scalable Spark Structured Streaming for REST API Destinations". A great article focusing on continuous Spark Structured Streaming (SSS). About a year old. I then decided, given customer demands to wo...
- 3966 Views
- 3 replies
- 2 kudos
- 2 kudos
Hi @MichTalebzadeh, Thank you for sharing your insightful article, "Building an Event-Driven Real-Time Data Processor with Spark Structured Streaming and API Integration." Your approach to integrating Apache Spark™, REST API, and Flask to create a ro...
- 2 kudos
- 1309 Views
- 0 replies
- 0 kudos
stored procedures
The plan for stored procedures in databricks spark has been announced in a few places. How can stored procedures look in Spark SQL?
- 1309 Views
- 0 replies
- 0 kudos
