Hey @AdamLucas !
Based on my personal experience, I can say that there are several practical areas that are more likely to be well-represented in the test. First, to understand and effectively operate Delta Lake, one has to have at least some general knowledge of its inner workings and transaction logs. While the ability to use ACID transactions is almost self-explanatory, the exam covers other aspects of Delta Lake in great detail. For example, you need to know the implications of the VACUUM command, what optimization strategies are best suited for different data sets, and how the data retention period affects the log in _delta_log.
The second practical area that I found challenging was structured streaming and ingestion. While the overall concept is straightforward, the specifics of the cloudFiles format and its differences when used in directory listing or file notification modes can be confusing. In my opinion, the most confusing part concerns the underlying mechanisms of schema inference and evolution and how exactly Append and Complete modes work.
Learning the specifics of how Spark works internally constitutes one more challenge. If you are not familiar with Sparkโs inner workings on a deep level, it is easy to confuse what runs on the Driver and what executes on the Workers. In addition, you need to know which operations Spark marks as wide transformations that cause the need for shuffling to optimize the query plan using the execution engine.
The most confusing aspect of Spark is Adaptive Query Execution capability that changes query plans while executions to improve performance. Although advanced tuning of such processes is outside the scope of the Data Engineer Associate certification, this topic is crucial to prepare for the Professional-level exam.
Finally, when working with orchestration, the main challenge was to plan the most reliable method of recovery from a task failure. When building a production-grade orchestration, one must plan how the entire system will recover from an unexpected event. For example, in the case of a failed task in Databricks, I should use the repair run option to relaunch the entire job from the point of failure without reprocessing all preceding tasks. Fortunately, I found that preparing for such details using the โdocs.databricks.comโ resources was much easier than trying to guess all the possible edge cases.
However, I have to say that I approached preparing for the exam in a way similar to any technical assessment. With a week or so before the test, I tried to spend as much time as possible practicing in a free community edition along with reading the official documentation and, eventually, went through a few unofficial practice tests from the major online learning platforms to see how well prepared I was.
All the very best towards your certification journey!!
