cancel
Showing results for 
Search instead for 
Did you mean: 
Data Engineering
Join discussions on data engineering best practices, architectures, and optimization strategies within the Databricks Community. Exchange insights and solutions with fellow data engineers.
cancel
Showing results for 
Search instead for 
Did you mean: 

How to write a Delta Live Table(dlt) pipeline output to Databricks SQL directly

VVill_T
Contributor

Hi,

I am trying to see if it is possible to setup a direct connection from dlt pipeline to a table in Databricks SQL by configuring the Target Schema:

image with poc being a location of schema like "dbfs:/***/***/***/poc.db

The error message was just a syntax error and I am not sure if I am missing a trick here:

imageI have read the documentation on Target Schema but I dont think it is related to Databricks SQL and I wonder if someone can point me to the right direction or advice me on if this could be done at all?

Many thanks in advance!

4 REPLIES 4

User16752244991
New Contributor III

Not quite sure what you mean by "output to Databricks SQL directly". DLT currently cannot write to Unity Catalog. The target schema name you specify in DLT pipeline settings currently becomes a Hive Metastore database and it will be created by DLT if it doesn't exist.

It sounds like what you are trying to do is:

You have an existing HMS database named "poc" with location set to "dbfs:/..../poc.db" and you would like to use this database as the target schema in your DLT pipeline. I see no reason why you shouldn't be able to do this. Based on the error message, it looks like either the database path is incorrect or it has special characters that are not allowed.

youssefmrini
Honored Contributor III

When ever you store a Delta Table to Hive Metastore. This table will be available in Databricks SQL Workspace ( Data Explorer ) under hive_metastore catalog.

It wont let me write to Hive Metastore but DBFS only - is that something I need to talk to my admin?

Rishabh-Pandey
Esteemed Contributor

You mean to say that you are not able to write in a hive metastore as well , in that case you need to ask to your admin to take a write permission ​

Rishabh Pandey

Connect with Databricks Users in Your Area

Join a Regional User Group to connect with local Databricks users. Events will be happening in your city, and you won’t want to miss the chance to attend and share knowledge.

If there isn’t a group near you, start one and help create a community that brings people together.

Request a New Group