What helped you the most when you were getting started with Databricks?

[edonaire]

Hi everyone,

I’m currently deepening my Databricks learning journey and one thing I’ve noticed is that there are many ways to get started, but not all of them help equally once you move from theory to real projects.

For those who already have some hands-on experience, what helped you the most in the beginning?

For example:

• SQL first or PySpark first?
• Best way to understand Unity Catalog in practice
• How to think about medallion architecture without overcomplicating it
• What beginner mistakes to avoid in notebooks, jobs, or pipelines
• Which features are most important to learn early for real business use cases

I’m especially interested in learning paths that go beyond isolated tutorials and help build real-world thinking.

Would love to hear what clicked for you when you were starting out.

Thanks!

[emma_s]

Hi,

For me I started from a SQL base but I really think it depends what background your coming from. If you are used to using Python day in day out then use that, if you've used SQL then use SQL first. Next I'd say understand declarative pipelines and lakeflow connect. Have a play and build out an end to end ingestion pipeline. This will also help you to think about medalion architecture and what you should do at each step in the architecture. Think about the use cases for materialized views vs streaming tables.
Next up, build a dashboard and a genie room. Finally, start experimenting with the AI functionality, it's good to put some PDFs into a volume and build a simple knowledge assistant you can then pair this up with your Genie room and work out which is best for what. 

At this point you'll start to understand the breadth of the Databricks platform, then you can start to focus in on one area.

Thanks,

Emma

[edonaire]

Hi Emma,

This is incredibly helpful, thank you for taking the time to share such a structured path.

I really like your point about starting from your background, it makes a lot of sense instead of forcing a one size fits all approach. The suggestion to build an end to end ingestion pipeline early also resonates a lot, especially to truly understand medallion architecture in practice rather than just conceptually.

The part about comparing materialized views vs streaming tables is something I have not explored deeply yet, so I will definitely look into that next. Also, bringing Genie rooms and AI use cases into the journey is a great perspective, it helps connect the platform beyond just data engineering.

Out of curiosity, when you mentioned focusing on one area later on, do you usually see people gravitating more towards data engineering, analytics, or AI within Databricks?

Thanks again, really appreciate it.

[emma_s]

Hi,

I'm glad it was helpful. I think people gravitate to their skillset in Databricks. So we tend to see statisticians and data scientists gravitating to the AI/ML and some basic data engineering. Whereas Data Engineers will stay in the Jobs and pipelines sections with some Unity catalogue and a bit of ML/AI ops stuff thrown in. Usually, I think it's driven by business need, existing skillset, and personal interests. Team structure is also a big player, some startups will have a few people doing everything, where as a big enterprise organisation will have seperate teams for each individual bit of the data pipeline.

Thanks,

Emma