cancel
Showing results for 
Search instead for 
Did you mean: 
Data Engineering
Join discussions on data engineering best practices, architectures, and optimization strategies within the Databricks Community. Exchange insights and solutions with fellow data engineers.
cancel
Showing results for 
Search instead for 
Did you mean: 

DLT table not picked in python notebook

Mathias_Peters
Contributor

Hi, 

I am a bit stumped atm bc I cannot figure out how to get a DLT table definition picked up in a Python notebook. 

1. I created a new notebook in python

2. added the following code: 

 

%python
import dlt
from pyspark.sql.functions import *

@dlt.table(
    comment="Some simple test",
    name="trx_dlt"
)
def transactions_live_table():
    df = spark.read.json("s3://FOLDER_LOCATION/Transaction/")
    return df

 

3. I created a new DLT pipeline and linked the Notebook. Linking did work bc the Notebook UI now allows me to validate my pieline code.

4. I press play and get 

   a. a warning the python magic command is not supported: 

Magic commands (e.g. %py, %sql and %run) are not supported with the exception of %pip within a Python notebook. Cells containing magic commands are ignored. Unsupported magic commands were found in the following notebooks

/Users/mathias.peters@epicompany.eu/DLT-TRX: %python

  b. an error that no DLT tables were discovered in the code: 

Mathias_Peters_0-1715334658498.png

 

How does that work? Where is my mistake here?

 

Kind regards, 

Mathias

 

 

1 ACCEPTED SOLUTION

Accepted Solutions

Mathias_Peters
Contributor

Ok, it seems that the default language of the notebook and the language of a particular cell can clash. If the default is set to Python, switching a cell to SQL won't work in DLT and vice versa. This is super unintuitive tbh.

View solution in original post

2 REPLIES 2

Mathias_Peters
Contributor

Update: this seems to be an issue with the notebook type and the magic commands at the beginnning. I have created a new DLT pipeline without attaching a notebook to it. That triggered the creation of an empty Python notebook. Switching this to SQL added the %sql command. Running the pipeline produced the same no tables discovered error as above. What is the correct way of specifying the notebook type here? (super unintuitive btw) 

Mathias_Peters
Contributor

Ok, it seems that the default language of the notebook and the language of a particular cell can clash. If the default is set to Python, switching a cell to SQL won't work in DLT and vice versa. This is super unintuitive tbh.

Join 100K+ Data Experts: Register Now & Grow with Us!

Excited to expand your horizons with us? Click here to Register and begin your journey to success!

Already a member? Login and join your local regional user group! If there isn’t one near you, fill out this form and we’ll create one for you to join!