- Mark as New
- Bookmark
- Subscribe
- Mute
- Subscribe to RSS Feed
- Permalink
- Report Inappropriate Content
12-19-2024 03:01 AM
Hi,
I have several different cells in my notebook that are connected to a DLT pipeline. Why are some cells skipped and others aren't?
I get the message "cannot run a cell when connected to the pipeline Databricks" when try running a cell when I'm connected to the pipeline compute and not a serverless/all-purpose cluster. Why does this happen? The goal is to utilize the cells in the notebook that aren't being run because those cells are refreshing my Power BI workspace.
- Mark as New
- Bookmark
- Subscribe
- Mute
- Subscribe to RSS Feed
- Permalink
- Report Inappropriate Content
12-19-2024 03:40 AM
Hi, @mkEngineer
When working with Delta Live Tables (DLT) in Databricks, you cannot run individual cells interactively as you would in a standard Databricks notebook.
DLT Pipeline Behavior:
- Delta Live Tables notebooks are executed as part of a managed pipeline. When you attach your notebook to a DLT pipeline, the execution of the cells is managed entirely by the pipeline runtime.
- The pipeline runtime does not allow arbitrary execution of cells. It executes only the cells required to define the tables, views, or streams specified in the DLT configurations (@dlt.table, @dlt.view, etc.).
- Cells that do not contribute directly to the DLT pipeline, much like magic commands (e.g., %py, %sql, and %run), are ignored by the runtime, with the exception of %pip within a Python notebook.
Takuya Omi (尾美拓哉)
