Future of Movie Discovery: How I Built an AI Movie Recommendation Agent on Databricks Free Edition

[Brahmareddy]

As a data engineer deeply passionate about how data and AI can come together to create real-world impact, I’m excited to share my project for the Databricks Free Edition Hackathon 2025 — Future of Movie Discovery (FMD). Built entirely on Databricks Free Edition, this project represents my belief that great innovation doesn’t always need massive infrastructure — it just needs the right platform and curiosity to explore possibilities.

Future of Movie Discovery (FMD) is an AI-powered recommendation agent that helps users find movies based on their mood and natural-language input. Instead of endless scrolling, users can simply say, “I want something thrilling,” or “Give me a relaxing comedy,” and the system instantly finds relevant titles from the Netflix Movies dataset. Behind the scenes, it’s powered by PySpark transformations, SentenceTransformer embeddings, and Delta Tables — all orchestrated seamlessly within the Databricks Lakehouse environment.

The project follows a complete end-to-end flow: data ingestion, cleaning, embedding generation, semantic similarity matching, and recommendation delivery through an interactive UI. I designed it to mirror a production-ready architecture, but with the agility and simplicity of Databricks Free Edition. The embeddings were generated using the all-MiniLM-L6-v2 model to capture the essence of each movie description and store them in Delta Tables for vector-based querying. When a user enters a prompt, the system dynamically encodes it, finds the closest matching embeddings, and returns movie recommendations that align with both genre and emotion.

To make the experience more engaging, I added a Mood Selector that lets users choose between Happy, Relaxed, Thoughtful, and Intense modes — and a Conversational Memory feature that remembers past searches for a more personal, agent-like experience. It’s not just a recommender; it’s a glimpse into how intelligent systems can adapt to human emotions through data.

What makes this special for me is not just the outcome, but the process. Working entirely within Databricks Free Edition, I was able to go from idea to a working AI agent without needing complex infrastructure or costly compute. It showed me once again how Databricks empowers data engineers like us to experiment, build, and innovate faster — uniting data engineering, AI, and creativity in one workspace.

The live version of this project is available here: https://fmd-ai.teamdataworks.com.
You can also watch my 5-minute demo here: YouTube – Future of Movie Discovery: AI + Data in Action 

This project marks the beginning of many more Data + AI experiments under my brand Team Data Works — where I aim to keep building solutions that merge data discipline with human intuition. My goal is to inspire more engineers to dream big, start small, and prove that with the right tools, even a single notebook can spark something meaningful.

[hasnat_unifeye]

Hi @Brahmareddy ,

Really enjoyed your hackathon demo. you’ve set a high bar for NLP-focused projects. I picked up a lot from your approach and it’s definitely given me ideas to try out.

For my hackathon entry, I took a similar direction using pyspark.ml.clustering with an LDA topic modelling approach to drive recommendations via clustering. In my case, I categorised recipes into topics based on word frequency and distribution across a large dataset. That gives each recipe a topic “label”, which can then be used to recommend similar recipes based on their semantic profile. 

If you’re interested, you can check it out here:
https://www.youtube.com/watch?v=JX0qyBD7qyM

[Brahmareddy]

Hi @hasnat_unifeye,

Thanks so much for the kind words! I really appreciate it — your approach sounds amazing too. Using LDA with PySpark ML for topic-based recommendations is a clever and explainable way to group similar items. I love how you’ve applied it to recipes — that’s such a relatable and data-rich use case.

I completely agree that projects like these show how versatile Databricks can be for NLP and semantic exploration. Would love to stay connected and exchange ideas — I think there’s a lot of overlap between clustering approaches and embedding-based similarity. Great work on your entry!

[AlbertaBode]

Really cool project! The mood-based movie matching and conversational memory make the whole discovery experience feel way more intuitive. It’s interesting because most people still browse platforms manually — like on streaming App — but your system shows how much smarter and personalized the future of content discovery can be. Impressive work!

[Brahmareddy]

Hi AlbertaBode,

Thank you so much for the kind words!
That’s exactly what I hoped to achieve with Future of Movie Discovery — making content exploration feel natural and intelligent, rather than static browsing. The mood-based matching and conversational memory were inspired by how people actually decide what they feel like watching.
Your feedback truly means a lot — it keeps me motivated to push the boundaries of how Data + AI can make everyday experiences smarter and more human.

[ashford]

Das ist ein beeindruckendes Projekt, besonders die Kombination aus Stimmungsanalyse, Embeddings und personalisierten Filmempfehlungen. Xtream Codes bestehen aus Server Adresse, Benutzername und Passwort für deinen IPTV Zugang. Wir zeigen dir, wie du die Daten in Apps wie IPTV Smarters oder TiviMate einträgst.

[juanlozadab]

Really nice end-to-end project. I like that you kept the whole flow inside the Lakehouse, from the PySpark transformations and Delta tables all the way to embedding-based retrieval and the UI.

The architecture also makes good use of the platform for a relatively lightweight use case. Storing the embeddings in Delta and keeping the recommendation flow close to the data is a clean approach, especially for a Free Edition project.

Nice example of how data engineering and AI can be combined without adding unnecessary infrastructure.

[Brahmareddy]

Thanks, @juanlozadab jlb. I really appreciate you digging into the architecture and not just the demo.

Keeping embeddings in Delta right next to the source tables was actually the part I went back and forth on most. It was tempting to reach for a dedicated vector store, but for this scale it felt like unnecessary infrastructure for infrastructure's sake, and staying in the Lakehouse meant one governance model and one place to debug when retrieval quality looked off.

Free Edition made that constraint easy to respect too and no budget to justify a separate service, so the simplest path was also the right one.

Curious if you've tried a similar embeddings-in-Delta pattern at a larger scale and I'd guess the tradeoffs start looking different once you're past a few million rows.

Thanks again!

[juanlozadab]

@Brahmareddy 
I haven't used this exact pattern at multi-million-row scale, but I agree that the tradeoffs start to change as the dataset and query volume grow.

I would probably keep Delta as the source of truth and separate the serving layer based on the access pattern. If the main requirement is semantic retrieval over data already living in Delta, AI Search with a Delta Sync Index feels like the natural evolution.

For an application like this one, though, Lakebase is also interesting. Since you already have conversational memory and application state, Lakebase could handle that operational side while also supporting vector search through Lakebase Search. That could be a good fit if you want low-latency application data and retrieval closer together.

So I think your current design makes a lot of sense for this scale. I would only introduce one of those additional serving layers once the access patterns or latency requirements actually justify it.

[Brahmareddy]

Hi @juanlozadab , thanks for sharing this thoughtful perspective! Keeping Delta as the source of truth and choosing the serving layer based on the access pattern makes sense to me.

For this project, I focused on getting the complete experience working at a manageable scale. As it grows, I’d want to measure retrieval latency and query volume before adding another layer. Your point about Lakebase is especially useful given the conversational memory and application state. That gives me a good direction to explore for the next version. Appreciate the practical advice! Thanks Jib!