cancel
Showing results for 
Search instead for 
Did you mean: 
Community Articles
Dive into a collaborative space where members like YOU can exchange knowledge, tips, and best practices. Join the conversation today and unlock a wealth of collective wisdom to enhance your experience and drive success.
cancel
Showing results for 
Search instead for 
Did you mean: 

Forum Posts

xdx001
by New Contributor II
  • 602 Views
  • 0 replies
  • 1 kudos

Strong Databricks Fundamental - Gen Z

Why Databricks is the Future of Data Analytics for Gen ZIn the fast-paced world of data analytics, staying ahead of the curve is crucial. For Gen Z, who are digital natives and always on the lookout for the latest tech trends, understanding the diffe...

  • 602 Views
  • 0 replies
  • 1 kudos
MichTalebzadeh
by Valued Contributor
  • 2955 Views
  • 3 replies
  • 0 kudos

Financial Crime detection with the help of Apache Spark, Data Mesh and Data Lake

For those interested in Data Mesh and Data Lakes for FinCrime detection:Data mesh is a relatively new architectural concept for data management that emphasizes domain-driven data ownership and self-service data availability. It promotes the decentral...

Community Articles
data lakes
Data Mesh
financial crime
spark
  • 2955 Views
  • 3 replies
  • 0 kudos
Latest Reply
carrolbeau
New Contributor II
  • 0 kudos

It's great that you're focusing on financial crime detection with advanced technologies like Apache Spark, Data Mesh, and Data Lake. For those looking to dive deeper into criminal records and related data, tools like KY criminal lookup can provide es...

  • 0 kudos
2 More Replies
ThomazRossito
by Contributor
  • 3225 Views
  • 1 replies
  • 1 kudos

Post: Lakehouse Federation - Databricks

Lakehouse Federation - Databricks In the world of data, innovation is constant. And the most recent revolution comes with Lakehouse Federation, a fusion between data lakes and data warehouses, taking data manipulation to a new level. This advancement...

Community Articles
data engineer
Lakehouse
SQL Analytics
  • 3225 Views
  • 1 replies
  • 1 kudos
Latest Reply
Freshman
New Contributor III
  • 1 kudos

Hey Quick Question, Can we use it for the production version ? We have application server as SQL server, we are planning to use lakehouse federation so we can bypass creating and maintaining 100 of workflows. as we a small dataset I am not too sure o...

  • 1 kudos
Shahram
by New Contributor II
  • 605 Views
  • 0 replies
  • 1 kudos

Hub Star Modeling 2.0 for Medalion Architecture

Excited to share my latest publication on arXiv!“Hub Star Modeling 2.0 for Medallion Architecture” https://arxiv.org/abs/2504.08788This new version builds on the original Hub Star Modeling approach, published last year, and now tailored for the Meda...

  • 605 Views
  • 0 replies
  • 1 kudos
genevive_mdonça
by Databricks Employee
  • 1645 Views
  • 1 replies
  • 6 kudos

Handling Complex Nested JSON in Databricks Using schemaHints

When I first got into managing schemas in Databricks, it took me a while to realize that putting in a little planning up front could save me a ton of headaches later on.I was working with these deeply nested, constantly changing JSON files. At first,...

  • 1645 Views
  • 1 replies
  • 6 kudos
Latest Reply
Advika
Databricks Employee
  • 6 kudos

Great tip @genevive_mdonça! schemaHints help avoid issues with evolving JSON data, making data processing more reliable and easier to maintain. Thanks for sharing.

  • 6 kudos
techgeorge
by New Contributor II
  • 1046 Views
  • 1 replies
  • 0 kudos

Understanding Coalesce, Skewed Joins, and Why AQE Doesn't Always Intervene

In Spark, data skew can be the silent killer of performance. One wide partition pulling in 90% of the data?But even with AQE (Adaptive Query Execution) turned on in Databricks, skewness isn't always automatically identified— and here’s why.What Is co...

Data Skew.png
  • 1046 Views
  • 1 replies
  • 0 kudos
Latest Reply
BigRoux
Databricks Employee
  • 0 kudos

@mark_ott , this question seems right up your alley. Care to comment?

  • 0 kudos
Yuki
by Contributor
  • 1351 Views
  • 0 replies
  • 1 kudos

One of the solution of [FAILED_READ_FILE.NO_HINT] Error while reading file, when display() or SELECT

One of the solution of [FAILED_READ_FILE.NO_HINT] Error while reading file, when display() or SELECTI got stuck with the above error when using `spark.read.table().display()` or directly query the table using %sql.While the display method is just one...

  • 1351 Views
  • 0 replies
  • 1 kudos
techgeorge
by New Contributor II
  • 596 Views
  • 0 replies
  • 0 kudos

How to train a Convolutional Neural Network on Databricks with Tensorflow and Keras

Here is how to trained a lightweight Convolutional Neuronal Network (CNN) to detect pneumonia from chest X-rays pictures on Azure Databricks. I promise no LLMs, no hype, just real-world deep learning:1. Built it with TensorFlow & Keras on Databricks2...

techgeorge_0-1743756172384.png
  • 596 Views
  • 0 replies
  • 0 kudos
shubham_meshram
by New Contributor II
  • 1099 Views
  • 0 replies
  • 0 kudos

When Did the Data Go Wrong? Using Delta Lake Time Travel for Investigation in Databricks

I. IntroductionData pipelines are the lifeblood of modern data-driven organizations. However, even the most robust pipelines can experience unexpected issues: data corruption, erroneous updates, or sudden data drops. When these problems occur, quickl...

shubham_meshram_0-1743459167949.png
  • 1099 Views
  • 0 replies
  • 0 kudos
Brahmareddy
by Esteemed Contributor
  • 1425 Views
  • 0 replies
  • 1 kudos

Use Query Patterns to Suggest Indexes Dynamically

Hey folks,Ever notice how a query that used to run super fast suddenly starts dragging? We’ve all been there. As data grows, those little inefficiencies in your SQL start showing up — and they show up hard. That’s where something cool comes in: using...

  • 1425 Views
  • 0 replies
  • 1 kudos
DataDarvish
by New Contributor II
  • 1395 Views
  • 0 replies
  • 1 kudos

Unit Testing for Data Engineering: How to Ensure Production-Ready Data Pipelines

In today’s data-driven world, the success of any business use case relies heavily on trust in the data. This trust is built upon key pillars such as data accuracy, consistency, freshness, and overall quality. When organizations release data into prod...

  • 1395 Views
  • 0 replies
  • 1 kudos

Join Us as a Local Community Builder!

Passionate about hosting events and connecting people? Help us grow a vibrant local community—sign up today to get started!

Sign Up Now
Labels