Issue in "Build a Declarative Pipeline with Spark Declarative Pipelines"

[seanpmcn]

I am trying to complete the "Get Started with Data Engineering" course, but I have been running into an issue.

I have gotten to this step in "Build a Declarative Pipeline with Spark Declarative Pipelines":

Demo: Create and Run the Pipeline

Now you'll configure an ETL Pipeline in the Databricks UI that uses this notebook as its source.

Follow these steps:

1. In the left sidebar, click Jobs & Pipelines (right-click → open in a new tab so you can refer back to these instructions)
2. Click Create and select ETL Pipeline
3. Give your pipeline a name (e.g., yourname-sdp-bronze-silver-gold)
4. Under Source code, click Browse and navigate to this notebook:
   • Find your project folder → select 08 - Build a Declarative Pipeline with Spark Declarative Pipelines
5. Under Destination, set:
   • Catalog: dbacademy
   • Schema: get_started_de
6. Under Compute, confirm Serverless is selected
7. Click Create

The problem is that the UI doesn't match the steps. Clicking "Create" automatically creates an empty pipeline, which is fine, but there doesn't seem to be anywhere to select source code.

Is there a workaround for this?

[seanpmcn · Accepted solution]

I have added the source now. I went to the root folder > Create pipeline asset > New pipeline source code folder. I was then able to go to the right of the pipeline name at the top of the screen and edit the catalogue and schema.

However, after running the pipeline, I received the error in the screenshot attached.

Reviewing the warning below the error, I believe I may just need to change the language from python to SQL and remove %sql from the code.

I will report back whether that works.

[seanpmcn]

It worked!

[Khasim_1]

Hi seanpmcn,

Great job troubleshooting that! It is very common for the UI steps in training documentation to occasionally lag behind the platform's rapid evolution, and you navigated that perfectly by exploring the "Create pipeline asset" workflow.

Your experience with the language syntax error is a perfect example of how declarative pipelines (like DLT/SDP) require strict environment settings compared to interactive notebooks. When a pipeline is configured for SQL, the engine expects pure SQL execution, and the inclusion of %sql magic commands (which are meant for interactive Python/notebook shells) will naturally trigger an error.

Architect’s Pro-Tip: When moving from interactive development (using %sql or %python cells) to declarative production (pipelines), I always recommend cleaning up the code into pure language-specific files or standard notebooks. It not only avoids these execution errors but also makes your code much more compatible with Databricks Asset Bundles (DABs) for CI/CD later on.

It’s great that you shared the solution—this will definitely save the next person.

[seanpmcn]

Thank you for the additional context and advice.

I have often seen jokes online about the frustration of looking online for help with an issue and finding a post from someone with the same problem, only for the original poster to reply saying they figured it out but not explaining how. So I am trying to avoid making that situation.

I think it's also good practice for ensuring proper documentation on a project and especially for helping oneself to learn and advance. I've had a history of running into the same issue multiple times and needing to discover the same solution over and over. It's tempting to move on after resolving a difficult issue, but I think it's worth it to do some root cause analysis.

[ahjimomo]

Tried and it works, for the script, I just updated the default language to `SQL`