Future of Movie Discovery: How I Built an AI Movie Recommendation Agent on Databricks Free Edition

[Brahmareddy]

As a data engineer deeply passionate about how data and AI can come together to create real-world impact, I’m excited to share my project for the Databricks Free Edition Hackathon 2025 — Future of Movie Discovery (FMD). Built entirely on Databricks Free Edition, this project represents my belief that great innovation doesn’t always need massive infrastructure — it just needs the right platform and curiosity to explore possibilities.

Future of Movie Discovery (FMD) is an AI-powered recommendation agent that helps users find movies based on their mood and natural-language input. Instead of endless scrolling, users can simply say, “I want something thrilling,” or “Give me a relaxing comedy,” and the system instantly finds relevant titles from the Netflix Movies dataset. Behind the scenes, it’s powered by PySpark transformations, SentenceTransformer embeddings, and Delta Tables — all orchestrated seamlessly within the Databricks Lakehouse environment.

The project follows a complete end-to-end flow: data ingestion, cleaning, embedding generation, semantic similarity matching, and recommendation delivery through an interactive UI. I designed it to mirror a production-ready architecture, but with the agility and simplicity of Databricks Free Edition. The embeddings were generated using the all-MiniLM-L6-v2 model to capture the essence of each movie description and store them in Delta Tables for vector-based querying. When a user enters a prompt, the system dynamically encodes it, finds the closest matching embeddings, and returns movie recommendations that align with both genre and emotion.

To make the experience more engaging, I added a Mood Selector that lets users choose between Happy, Relaxed, Thoughtful, and Intense modes — and a Conversational Memory feature that remembers past searches for a more personal, agent-like experience. It’s not just a recommender; it’s a glimpse into how intelligent systems can adapt to human emotions through data.

What makes this special for me is not just the outcome, but the process. Working entirely within Databricks Free Edition, I was able to go from idea to a working AI agent without needing complex infrastructure or costly compute. It showed me once again how Databricks empowers data engineers like us to experiment, build, and innovate faster — uniting data engineering, AI, and creativity in one workspace.

The live version of this project is available here: https://fmd-ai.teamdataworks.com.
You can also watch my 5-minute demo here: YouTube – Future of Movie Discovery: AI + Data in Action 

This project marks the beginning of many more Data + AI experiments under my brand Team Data Works — where I aim to keep building solutions that merge data discipline with human intuition. My goal is to inspire more engineers to dream big, start small, and prove that with the right tools, even a single notebook can spark something meaningful.

[hasnat_unifeye]

Hi @Brahmareddy ,

Really enjoyed your hackathon demo. you’ve set a high bar for NLP-focused projects. I picked up a lot from your approach and it’s definitely given me ideas to try out.

For my hackathon entry, I took a similar direction using pyspark.ml.clustering with an LDA topic modelling approach to drive recommendations via clustering. In my case, I categorised recipes into topics based on word frequency and distribution across a large dataset. That gives each recipe a topic “label”, which can then be used to recommend similar recipes based on their semantic profile. 

If you’re interested, you can check it out here:
https://www.youtube.com/watch?v=JX0qyBD7qyM

[Brahmareddy]

Hi @hasnat_unifeye,

Thanks so much for the kind words! I really appreciate it — your approach sounds amazing too. Using LDA with PySpark ML for topic-based recommendations is a clever and explainable way to group similar items. I love how you’ve applied it to recipes — that’s such a relatable and data-rich use case.

I completely agree that projects like these show how versatile Databricks can be for NLP and semantic exploration. Would love to stay connected and exchange ideas — I think there’s a lot of overlap between clustering approaches and embedding-based similarity. Great work on your entry!