https://community.databricks.com/t5/data-engineering/is-address-line-4-the-place-where-data-goes-to-die/m-p/152843#M53888 <P>I have never worked on this problem but based on previous posts from other community user i have come to know that fuzzy logic can help finding records that are most likely to be same or similar . Here are some links where this has been discussed i guess .&nbsp;</P><P><A href="https://www.databricks.com/solutions/accelerators/product-matching-with-ml" target="_blank">https://www.databricks.com/solutions/accelerators/product-matching-with-ml</A></P><P><A href="https://community.databricks.com/t5/data-engineering/fuzzy-text-matching-in-spark/td-p/29805" target="_blank">https://community.databricks.com/t5/data-engineering/fuzzy-text-matching-in-spark/td-p/29805</A>&nbsp;</P><P>&nbsp;</P> Wed, 01 Apr 2026 02:11:03 GMT pradeep_singh 2026-04-01T02:11:03Z https://community.databricks.com/t5/data-engineering/is-address-line-4-the-place-where-data-goes-to-die/m-p/151884#M53722 <P data-path-to-node="4">I’ve spent the last few years jumping between insurance, healthcare, and retail, and I’ve come to a very painful conclusion that we should never have let humans type their own addresses into a text box.&nbsp;<span class="lia-unicode-emoji" title=":beaming_face_with_smiling_eyes:">😁</span></P> <P data-path-to-node="5">For a pet project, I’m currently looking at a dataset where the "City" column is a ghost town, but "Address Line 4" is a chaotic party of cities and counties. It’s making geographical analysis for healthcare accessibility basically impossible. You’re trying to map out deprived areas to actually help people, but the data is so scrambled it looks like everyone lives in a different dimension.</P> <P data-path-to-node="6">Retail seems to be the absolute Wild West of this. People just put things in whatever order their heart desires at 11 PM while buying shoes.</P> <P data-path-to-node="7">I’m curious to know how the rest of you are actually surviving this in your Databricks pipelines. Are we all just suffering through thousand-line SQL CASE statements that we update every time a new typo is invented? Or has anyone actually found a "magic bullet" tool or API that plays nice with Spark and doesn't cost a fortune? I’ve thought about throwing an LLM at it, but I’m worried it’ll just hallucinate everyone onto a private island.</P> <P data-path-to-node="8">What’s the most "creative" address you’ve had to clean up lately?</P> <P data-path-to-node="8">&nbsp;</P> Tue, 24 Mar 2026 19:45:44 GMT https://community.databricks.com/t5/data-engineering/is-address-line-4-the-place-where-data-goes-to-die/m-p/151884#M53722 Ashwin_DSA 2026-03-24T19:45:44Z https://community.databricks.com/t5/data-engineering/is-address-line-4-the-place-where-data-goes-to-die/m-p/152843#M53888 <P>I have never worked on this problem but based on previous posts from other community user i have come to know that fuzzy logic can help finding records that are most likely to be same or similar . Here are some links where this has been discussed i guess .&nbsp;</P><P><A href="https://www.databricks.com/solutions/accelerators/product-matching-with-ml" target="_blank">https://www.databricks.com/solutions/accelerators/product-matching-with-ml</A></P><P><A href="https://community.databricks.com/t5/data-engineering/fuzzy-text-matching-in-spark/td-p/29805" target="_blank">https://community.databricks.com/t5/data-engineering/fuzzy-text-matching-in-spark/td-p/29805</A>&nbsp;</P><P>&nbsp;</P> Wed, 01 Apr 2026 02:11:03 GMT https://community.databricks.com/t5/data-engineering/is-address-line-4-the-place-where-data-goes-to-die/m-p/152843#M53888 pradeep_singh 2026-04-01T02:11:03Z