topic Re: ⭐ Setup Spark with Hadoop Anywhere : A DBR aligned local Spark+HDFS+Hive stack on Docker⭐ in Announcements

⭐ Setup Spark with Hadoop Anywhere : A DBR aligned local Spark+HDFS+Hive stack on Docker⭐

K_Anudeep — Fri, 14 Nov 2025 12:16:07 GMT

Hello Community,

Let me start off with a quick question:

Have you ever...

Migrated your workloads from on-prem Spark to Databricks and encountered a bug and thought, “I wish I could repro this locally to debug the issue without burning cluster hours?”
Seen a third-party library work fine on OSS Spark but fail on Databricks, with no easy way to debug?
Needed to test behaviour across different Spark / Scala / Java versions… but spinning up multiple clusters just for that felt like overkill?
Want to set up a Spark environment locally to learn and experiment with the latest versions of Spark?

If yes, this project is literally built for you.

------------------------------------------------------------------------------------------

Meet Spark with Hadoop Anywhere

I’ve open-sourced a project called Spark with Hadoop Anywhere – a production-like Spark + Hadoop + Hive stack on Docker, specifically designed for:

On-prem Open source Spark and Databricks users (Support, SREs, Data Eng etc.)
People are debugging issues with DBR vs OSS Spark
Anyone who wants a realistic local analytics node can run it anywhere Docker is available
Anyone who wants to learn Spark and have a cluster-like setup installed locally

🔗 Project page (docs + overview):
https://anudeepkonaboina.github.io/spark-with-hadoop-anywhere/

🔗 GitHub repo (please ⭐ and fork!):
https://github.com/AnudeepKonaboina/spark-with-hadoop-anywhere

Key features (why you should at least fork it 👇)

1. DBR-aligned version matrix

Each Git branch is pinned to a specific Spark / Scala / Java combo, aligned with the OSS Spark versions used by Databricks Runtime (DBR).

More Details here: https://anudeepkonaboina.github.io/spark-with-hadoop-anywhere/#dbr-underlying-spark-oss-compatible-b...

2. Full analytics node, not just Spark

You don’t just get a Spark binary thrown in a container.

You get a single-node analytics stack:

Spark (standalone: master + worker in one container)
HDFS (namenode + datanode, real filesystem semantics)
Hive Metastore backed by PostgreSQL in a separate container
Hive CLI & Beeline wired up

3. One-command setup via `setup-spark.sh`

The entire stack is orchestrated through a single script, and just by running a single command, you will have a single-node cluster-like setup on your laptop

Why I’m posting this on Databricks Community

This project was born out of exactly the kind of pain users working with OSS spark or Databricks face:

Support / SRE / PS engineers needing fast, realistic repros
Customers hitting weird corner cases in Spark / Delta/connectors
Devs who want DBR-aligned OSS Spark locally without owning infrastructure

If that sounds like you, I’d honestly love it if you:

Fork the repo
Spin up the stack for the Spark version you care about
Try reproducing one of your current/old issues
Share feedback/issues / PRs

---------------------------------

Interested??

👉 Fork and explore it & star the repo if you like it:
https://github.com/AnudeepKonaboina/spark-with-hadoop-anywhere

👉 Docs/overview (easier to share inside your team):
https://anudeepkonaboina.github.io/spark-with-hadoop-anywhere/

If you end up using this to debug a challenging Spark/HDFS/Hive or DBR issue, please leave a comment or open an issue in the repository – I’d love to hear about your experience and what would make the stack even more useful for the Databricks community.

Re: ⭐ Setup Spark with Hadoop Anywhere : A DBR aligned local Spark+HDFS+Hive stack on Docker⭐

KaushalVachhani — Fri, 14 Nov 2025 03:25:22 GMT

This is amazing, @K_Anudeep. Users will benefit from this.

Re: ⭐ Setup Spark with Hadoop Anywhere : A DBR aligned local Spark+HDFS+Hive stack on Docker⭐

JAHNAVI — Fri, 14 Nov 2025 03:52:53 GMT

Fantastic @K_Anudeep, this is truly amazing, and this will help a lot and provide such a lightweight environment.

Re: ⭐ Setup Spark with Hadoop Anywhere : A DBR aligned local Spark+HDFS+Hive stack on Docker⭐

nikhilj0421 — Fri, 14 Nov 2025 07:19:35 GMT

This is fantastic @K_Anudeep, and really helpful.

Re: ⭐ Setup Spark with Hadoop Anywhere : A DBR aligned local Spark+HDFS+Hive stack on Docker⭐

RevanthV — Fri, 14 Nov 2025 11:14:59 GMT

Hey, I have tried this out on my laptop and it hardly takes 3 minutes to setup a cluster like env locally..This is really helpful.Great work @K_Anudeep 👏👏

Re: ⭐ Setup Spark with Hadoop Anywhere : A DBR aligned local Spark+HDFS+Hive stack on Docker⭐

Rajasaiharish — Wed, 19 Nov 2025 04:39:34 GMT

I have been finding ways to get a cluster like env to explore spark 4 and this is quite helpful.Thanks @Anudeep Which helped in resolving a production bug and we dont any sandbox environment to check and we dont need to waste the time in set up environment Thanks for this

Re: ⭐ Setup Spark with Hadoop Anywhere : A DBR aligned local Spark+HDFS+Hive stack on Docker⭐

User16593473949 — Wed, 19 Nov 2025 06:40:38 GMT

This is really amazing, @K_Anudeep. It will be very helpful for testing and debugging. Kudos to you.

Re: ⭐ Setup Spark with Hadoop Anywhere : A DBR aligned local Spark+HDFS+Hive stack on Docker⭐

Raman_Unifeye — Wed, 19 Nov 2025 09:05:11 GMT

Very helpful for local testing.

Re: ⭐ Setup Spark with Hadoop Anywhere : A DBR aligned local Spark+HDFS+Hive stack on Docker⭐

BS_THE_ANALYST — Wed, 19 Nov 2025 11:02:44 GMT

This is freaking awesome @K_Anudeep, definitely going in the favourites.

Can't wait to test this out. Big kudos to you, sir.

All the best,
BS

Re: ⭐ Setup Spark with Hadoop Anywhere : A DBR aligned local Spark+HDFS+Hive stack on Docker⭐

KyungjunLee — Tue, 25 Nov 2025 08:09:15 GMT

cool

Re: ⭐ Setup Spark with Hadoop Anywhere : A DBR aligned local Spark+HDFS+Hive stack on Docker⭐

analyticsnerd — Tue, 25 Nov 2025 13:52:03 GMT

This is really good ....Just tried it feels like getting a complete platform for analytics in my laptop

Neatly done, 💯💯 @K_Anudeep

Re: ⭐ Setup Spark with Hadoop Anywhere : A DBR aligned local Spark+HDFS+Hive stack on Docker⭐

Ram_123 — Wed, 26 Nov 2025 06:29:37 GMT

This is super cool!! I will try it out soon 😃but the documentation is commendable @K_Anudeep

Re: ⭐ Setup Spark with Hadoop Anywhere : A DBR aligned local Spark+HDFS+Hive stack on Docker⭐

siva-anantha — Wed, 03 Dec 2025 20:10:04 GMT

Thank you for sharing this. I have tried this in the past and gave up at certain point. It is helpful

Re: ⭐ Setup Spark with Hadoop Anywhere : A DBR aligned local Spark+HDFS+Hive stack on Docker⭐

shahbaigmcp — Thu, 11 Dec 2025 23:38:30 GMT

Thank you so much for sharing @K_Anudeep

topic Re: ⭐ Setup Spark with Hadoop Anywhere : A DBR aligned local Spark+HDFS+Hive stack on Docker⭐ in Announcements

⭐ Setup Spark with Hadoop Anywhere : A DBR aligned local Spark+HDFS+Hive stack on Docker⭐

Meet Spark with Hadoop Anywhere

Key features (why you should at least fork it 👇)

1. DBR-aligned version matrix

2. Full analytics node, not just Spark

3. One-command setup via setup-spark.sh

Why I’m posting this on Databricks Community

Interested??

Re: ⭐ Setup Spark with Hadoop Anywhere : A DBR aligned local Spark+HDFS+Hive stack on Docker⭐

Re: ⭐ Setup Spark with Hadoop Anywhere : A DBR aligned local Spark+HDFS+Hive stack on Docker⭐

Re: ⭐ Setup Spark with Hadoop Anywhere : A DBR aligned local Spark+HDFS+Hive stack on Docker⭐

Re: ⭐ Setup Spark with Hadoop Anywhere : A DBR aligned local Spark+HDFS+Hive stack on Docker⭐

Re: ⭐ Setup Spark with Hadoop Anywhere : A DBR aligned local Spark+HDFS+Hive stack on Docker⭐

Re: ⭐ Setup Spark with Hadoop Anywhere : A DBR aligned local Spark+HDFS+Hive stack on Docker⭐

Re: ⭐ Setup Spark with Hadoop Anywhere : A DBR aligned local Spark+HDFS+Hive stack on Docker⭐

Re: ⭐ Setup Spark with Hadoop Anywhere : A DBR aligned local Spark+HDFS+Hive stack on Docker⭐

Re: ⭐ Setup Spark with Hadoop Anywhere : A DBR aligned local Spark+HDFS+Hive stack on Docker⭐

Re: ⭐ Setup Spark with Hadoop Anywhere : A DBR aligned local Spark+HDFS+Hive stack on Docker⭐

Re: ⭐ Setup Spark with Hadoop Anywhere : A DBR aligned local Spark+HDFS+Hive stack on Docker⭐

Re: ⭐ Setup Spark with Hadoop Anywhere : A DBR aligned local Spark+HDFS+Hive stack on Docker⭐

Re: ⭐ Setup Spark with Hadoop Anywhere : A DBR aligned local Spark+HDFS+Hive stack on Docker⭐

3. One-command setup via `setup-spark.sh`