https://community.databricks.com/t5/data-engineering/how-to-use-partial-parse-msgpack-with-workflow-dbt-task/m-p/14224#M8751 <P>I'm looking for direction on how to get the dbt task in workflows to use the partial_parse.msgpack file to skip parsing files that haven't changed. I'm downloading my artifacts after each run and the partial_parse file is being saved back to adls.</P><P></P><P>What is the dbt task doing? Which folder does the git repo specified in "project directory" load to? I'm assuming I can use an init script and load the file from adls and dump it into whichever folder is /{project directory}/target/ ?</P> Thu, 29 Dec 2022 02:23:20 GMT Jfoxyyc 2022-12-29T02:23:20Z https://community.databricks.com/t5/data-engineering/how-to-use-partial-parse-msgpack-with-workflow-dbt-task/m-p/14224#M8751 <P>I'm looking for direction on how to get the dbt task in workflows to use the partial_parse.msgpack file to skip parsing files that haven't changed. I'm downloading my artifacts after each run and the partial_parse file is being saved back to adls.</P><P></P><P>What is the dbt task doing? Which folder does the git repo specified in "project directory" load to? I'm assuming I can use an init script and load the file from adls and dump it into whichever folder is /{project directory}/target/ ?</P> Thu, 29 Dec 2022 02:23:20 GMT https://community.databricks.com/t5/data-engineering/how-to-use-partial-parse-msgpack-with-workflow-dbt-task/m-p/14224#M8751 Jfoxyyc 2022-12-29T02:23:20Z https://community.databricks.com/t5/data-engineering/how-to-use-partial-parse-msgpack-with-workflow-dbt-task/m-p/14226#M8753 <P>The use case is I'd like the dbt task in a databricks workflow to be able to use the partial_parse.msgpack file so it can take advantage of partial parsing and not have to parse files that haven't changed. It significantly speeds up runtime since it doesn't waste time parsing.</P><P></P><P>I download the file from the artifacts after each run and store it in an adls location. I don't know how to get the dbt task to use it. I don't want to store it in git repo. I would prefer it if I could get the dbt task to copy the file from adls somehow. If this was possible, we could also copy manifest.json and run state:modified tasks for slim ci.</P> Tue, 03 Jan 2023 08:50:25 GMT https://community.databricks.com/t5/data-engineering/how-to-use-partial-parse-msgpack-with-workflow-dbt-task/m-p/14226#M8753 Jfoxyyc 2023-01-03T08:50:25Z https://community.databricks.com/t5/data-engineering/how-to-use-partial-parse-msgpack-with-workflow-dbt-task/m-p/14225#M8752 <P>Hi, Could you please confirm what will be your expectation and the used case? Do you want the file to be saved somewhere else?</P><P></P> Mon, 02 Jan 2023 18:39:11 GMT https://community.databricks.com/t5/data-engineering/how-to-use-partial-parse-msgpack-with-workflow-dbt-task/m-p/14225#M8752 Debayan 2023-01-02T18:39:11Z