Importing python to DLT - Not working with DLT Pipeline

[AxelBrsn]

Hello, we are trying to adapt our developments (notebook with delta tables), into Delta Live Tables Pipelines.

We tried to import Python files that are very useful for data transformations (silver data cleaning, for example) :

• From the Cluster (run manually the declarative notebook of DLT) : It's working.
• From the DLT Pipeline : It's not, it can't find and reach the python files in question.

We tried to import it as Python files, and also as Python library (wheel file extension) :

• Images are available in attachment.

Thank you for your time, and your help!

[Kaniz]

Hi @AxelBrsn, When importing Python files, ensure that they are accessible to DLT Pipelines.

Here are some steps to troubleshoot:

• Spark Context Configuration: Create your Spark session with the necessary settings to import the Delta module. For example:

  spark_session = SparkSession.builder
      .master("local")
      .config("spark.jars.packages", "io.delta:delta-core_2.12:0.8.0")
      .getOrCreate()
  

  AI-generated code. Review and use carefully. More info on FAQ.
• Library Installation: As of now, DLT supports library installation via %pip. Package your useful Python code as a library and install it using %pip².
• Schema Definition: If you encounter issues related to the schema, ensure that you define the schema correctly. For example, use spark.readStream.format("cloudFiles").schema(sch) instead of .option("schema", sch)³

[AxelBrsn]

Hello @Kaniz, thanks for the answer, but it's not completely clear on my side.
I want to import, not to install ?

Why the import is not accessible from the DLT Pipeline?

[AxelBrsn · Accepted solution]

The solution is to import from Python but also add the python file in the Pipeline settings, in the list of source code.