Creating a DLT pipeline that reads from a JDBC source

Reda — Mon, 24 Oct 2022 17:33:59 GMT

Hey,

I'm trying to create a DLT pipeline that reads from a JDBC source, and the code I'm using looks something like this in python:

import dlt
@dlt.table
def table_name():
  driver = 'oracle.jdbc.driver.OracleDriver'
  url = '...'
  query = 'SELECT ... FROM ...'
  df = spark.read.format("jdbc")\
         .option("driver", driver)\
         .option("url",url)\
         .option("user", username)\
         .option("password", password)\
         .option("query", query)\
         .load()
  return df

it works perfectly fine when outside of the DLT pipeline, and I know for sure that the "df" DataFrame should be created successfully.

I keep getting an error in my DLT pipeline logs that fails at the "setting up tables" stage which says: " Failed to resolve flow: 'table_name' ".

I tried creating a basic DLT that just reads from an existing delta table (one that read from the same JDBC table but outside of the DLT pipeline) and it works fine, so I know that my environment setup is working fine.

Can anyone pinpoint what is going wrong here?

Re: Creating a DLT pipeline that reads from a JDBC source

Anonymous — Sun, 27 Nov 2022 14:18:38 GMT

Hi @Reda Bitar

Great to meet you, and thanks for your question!

Let's see if your peers in the community have an answer to your question first. Or else bricksters will get back to you soon.

Thanks

topic Creating a DLT pipeline that reads from a JDBC source in Data Engineering

Creating a DLT pipeline that reads from a JDBC source

Re: Creating a DLT pipeline that reads from a JDBC source