Hi guys,
15 days ago, I posted that I passed the Data Analyst cert: https://community.databricks.com/t5/community-articles/zero-to-hero-data-analysis-certification/m-p/...
Since then, I've only managed to study the first week of the content in the blended learning for Data Engineering (life gets busy!):
This weekend, mainly because I'm a nerd, I want to see if I can study the remaining Weeks 2, 3, and 4. If I'm feeling like a turbo-nerd, I'll squeeze in some practice papers or a mini-project; with that said, fresh air would also be a nice end to completing weeks 2,3,4 😲😏😂.
I thought I'd post this to motivate anyone who's on the fence about doing some studying ☺️.
All the best,
BS
Update:
Just finished Week 2: Deploy Workloads with Lakeflow Jobs
That was a really cool module. Love the fact that the tasks, within jobs, have so many options, most of the learning was centered around notebooks but you can orchestrate so many different tasks.
I'm looking forward to building pipelines with the different event triggers that I saw. There's file-event triggers, scheduled, and continuous. Definitely inspired me to build and end-to-end solution with a near-real-time refreshed dashboard ☺️.
Absolutely loving the facts that it's modular. If you have a job that consists of many tasks, you can then embed this job within another job. Jobs and tasks also support parameterisation - absolutely love this.
The "Repair Run" is also a really cool feature.
Can't wait to build out some projects with this. Now onto Week 3 learning today. For anyone else doing some weekend studying, best of luck.
All the best,
BS
I massively underestimated Week 3's content 🤣. Just got this one ticked off now, so I'll finish off week 4 throughout this week. Not as much as a "speedrun" as I anticipated. The week 3 content consisted of "Build Data Pipelines with Lakeflow Declarative Pipelines".
I've gotta say, I'm really really excited to use these. As I was going through the content I had loads of ideas and can see there's definitely a lot of mastery needed. An example, I setup a pipeline, but I wanted to alter a column's datatype on a streaming table 😔🤔. Not quite as easy as one would think. Got me thinking about how I'd then design an alternate solution.
Started making me think alot around backfilling tables, once they're in production. Also, what happens with _rescued_data? How would we cater for this etc.
The Expectations were a really cool find & I loved the Pipeline Event Logs. There's alot to consider with joining streaming tables as well. CDC was super cool with the APPLY CHANGES INTO although this seems to be replaced with AUTO CDC when I checked it via the docs: https://docs.databricks.com/aws/en/dlt/cdc . Also saw how everything can be executed through code, i.e. databricks asset bundles or SDK/CLI. We can also choose Python instead of SQL when building these things. So much to conquer 😈🤣.
There's a lot around DLT (declarative pipelines) so I'm excited to build some projects. The labs were a great help for getting me started. So yeah, DLT is not one for a speed run 🤣.
Till next week ☺️👀.
All the best,
BS
