- 5736 Views
- 3 replies
- 3 kudos
Resolved! Feature Engineering for Data Engineers: Building Blocks for ML Success
For a UK Government Agency, I made a Comprehensive presentation titled " Feature Engineering for Data Engineers: Building Blocks for ML Success". I made an article of it in Linkedlin together with the relevant GitHub code. In summary the code delve...
- 5736 Views
- 3 replies
- 3 kudos
- 3 kudos
This is a fantastic post! The detailed explanation of feature engineering, from handling missing values to using Variational Autoencoders (VAEs) for synthetic data generation, provides invaluable insights for improving machine learning models. The ap...
- 3 kudos
- 12502 Views
- 3 replies
- 6 kudos
Comprehensive Guide to Databricks Optimization: Z-Order, Data Compaction, and Liquid Clustering
Optimizing data storage and access is crucial for enhancing the performance of data processing systems. In Databricks, several optimization techniques can significantly improve query performance and reduce costs: Z-Order Optimize, Optimize Compaction...
- 12502 Views
- 3 replies
- 6 kudos
- 4813 Views
- 0 replies
- 0 kudos
How can Databricks AI/BI Genie, RAG, & LLMs seamlessly coexist with MS Copilot to drive innovation?
The future of enterprise productivity and analytics lies in the seamless integration of advanced tools like Databricks Genie AI/BI, RAG & LLMs and Microsoft Copilot. While each serves distinct purposes, their coexistence can unlock unparalleled value...
- 4813 Views
- 0 replies
- 0 kudos
- 1333 Views
- 0 replies
- 1 kudos
How Databricks Empowers Scalable Data Products Through Medallion Mesh Architecture?
Unlock the Power of Your Data: Solving Fragmentation and Governance Challenges!In today’s fast-paced, data-driven enterprises, fragmented data and governance issues create roadblocks to decision-making and innovation. Traditional architectures strugg...
- 1333 Views
- 0 replies
- 1 kudos
- 610 Views
- 0 replies
- 0 kudos
Rebuilding and Re-Platforming Your Databricks Lakehouse with Serverless Compute
Dear Databricks Community,In today’s fast-paced data landscape, managing infrastructure manually can slow down innovation, increase costs, and limit scalability. Databricks Serverless Compute solves these challenges by eliminating infrastructure over...
- 610 Views
- 0 replies
- 0 kudos
- 2156 Views
- 0 replies
- 3 kudos
Mapping Compliance Standards to Industries: A Comprehensive Guide
Brief Guideline: Mapping Compliance Standards to IndustriesThis guide provides a detailed mapping of various compliance standards to their respective industries, highlighting the specific sectors and descriptions for each standard. Understanding thes...
- 2156 Views
- 0 replies
- 3 kudos
- 2252 Views
- 3 replies
- 0 kudos
Getting data from Databricks into Excel using Databricks Jobs API
If you have your data in Databricks, but like to analyse it in Excel, you can use Web API on Power Query. It allows you to not just query an existing table, but also trigger the execution of a PySpark notebook using Databricks Jobs API, and get the d...
- 2252 Views
- 3 replies
- 0 kudos
- 0 kudos
Got it, yes you have specified the same in your message. Thanks for sharing.
- 0 kudos
- 1949 Views
- 1 replies
- 3 kudos
How to Grant Workspace Admin Permissions to an ID Using Parent Groups
Hello,There are several ways to grant Workspace Admin permissions in Databricks. While this may seem straightforward, I found it a bit confusing when I started using Databricks, so I’d like to share my experience. This guide is aimed at beginners.How...
- 1949 Views
- 1 replies
- 3 kudos
- 3 kudos
Thanks for sharing this is great!
- 3 kudos
- 632 Views
- 0 replies
- 0 kudos
Learn Data Engineering on Databricks step by step
For new aspiring Data Engineers, it has always been difficult to start their learning. With decade of experience in Data Engineering now I have put together a series of article that can help new aspirants. The list is small attempt to help new Data E...
- 632 Views
- 0 replies
- 0 kudos
- 2046 Views
- 3 replies
- 1 kudos
Is there any way to add a Matplotlib visualizaton to a notebook Dashboard?
So I love that databricks lets you display a dataframe, create a visualization of it, then add that visualization to notebook dashboard to present. However, the visualizations lack some customization that I would like. For example the heat map vis...
- 2046 Views
- 3 replies
- 1 kudos
- 1 kudos
This is correct, it seems the way you want to implement is not currently supported
- 1 kudos
- 2422 Views
- 0 replies
- 0 kudos
Unlock the Full Potential of Databricks with the Demo Center!
Hello Databricks community!If you're eager to explore how Databricks can revolutionize your data workflows, I highly recommend checking out the Databricks Demo Center. It’s packed with insights and tools designed to cater to both beginners and season...
- 2422 Views
- 0 replies
- 0 kudos
- 2813 Views
- 0 replies
- 0 kudos
Understanding Databricks Workspace IP Access List
What is a Databricks Workspace IP Access List?The Databricks Workspace IP Access List is a security feature that allows administrators to control access to the Databricks workspace by specifying which IP addresses or IP ranges are allowed or denied a...
- 2813 Views
- 0 replies
- 0 kudos
- 636 Views
- 0 replies
- 1 kudos
Python step-through debugger for Databricks Notebooks and Files is now Generally Available
Python step-through debugger for Databricks Notebooks and Files is now Generally Availablehttps://www.databricks.com/blog/announcing-general-availability-step-through-debugging-databricks-notebooks-and-files
- 636 Views
- 0 replies
- 1 kudos
- 1315 Views
- 1 replies
- 4 kudos
Orchestrate Databricks jobs with Apache Airflow
You can Orchestrate Databricks jobs with Apache AirflowThe Databricks provider implements the below operators:DatabricksCreateJobsOperator : Create a new Databricks job or reset an existing jobDatabricksRunNowOperator : Runs an existing Spark job run...
- 1315 Views
- 1 replies
- 4 kudos
- 4 kudos
Good one @Sourav-Kundu! Your clear explanations of the operators really simplify job management, plus the resource link you included makes it easy for everyone to dive deeper .
- 4 kudos
- 1266 Views
- 1 replies
- 2 kudos
Use Retrieval-augmented generation (RAG) to boost performance of LLM applications
Retrieval-augmented generation (RAG) is a method that boosts the performance of large language model (LLM) applications by utilizing tailored data.It achieves this by fetching pertinent data or documents related to a specific query or task and presen...
- 1266 Views
- 1 replies
- 2 kudos
- 2 kudos
Thanks for sharing such valuable insight, @Sourav-Kundu . Your breakdown of how RAG enhances LLMs is spot on- clear and concise!
- 2 kudos
