cancel
Showing results forย 
Search instead forย 
Did you mean:ย 
Data Engineering
Join discussions on data engineering best practices, architectures, and optimization strategies within the Databricks Community. Exchange insights and solutions with fellow data engineers.
cancel
Showing results forย 
Search instead forย 
Did you mean:ย 

Create a docker image for dbt task

essura
New Contributor II

Hi there,

We are trying to setup up a docker image for our dbt execution, primarily to improve execution speed, but also to simplify deployment (we are using a private repos for both the dbt project and some of the dbt packages).

It seems to work currently, but fx downloading artifacts is not working and there are some complaints in log4j surrounding missing R. 

Is it possible to see the built-in dbt task Dockerfile? or at least get an outline of used directories, env variables and dependencies?

/Esben

1 REPLY 1

essura
New Contributor II

This looks kind of what ChatGPT proposed as a solution to me.

My issue is that i'm trying to use our custom container in a dbt task (a workflow task). The container is working, but we can't fx use the "download artifact", probably because the path is configured to some temp execution path. And we have some runtime issues in the logs concerning missing R, but what is R used for in the dbt task?

 

Connect with Databricks Users in Your Area

Join a Regional User Group to connect with local Databricks users. Events will be happening in your city, and you wonโ€™t want to miss the chance to attend and share knowledge.

If there isnโ€™t a group near you, start one and help create a community that brings people together.

Request a New Group