topic Re: DAB deploy missing packages in Data Engineering

DAB deploy missing packages

Przemk00 — Tue, 22 Oct 2024 10:48:10 GMT

Hello everyone, not sure where to start but so will do it from the beginning.

I encountered an issue where my .whl files in the dist/ and packages/ directories were not being deployed to Databricks using dab-deploy. After investigating, I discovered that these directories were listed in .gitignore, which prevented them from being included in the deployment. As a workaround, commenting out the dist/ and packages/ entries in .gitignore ensures the deployment is successful.

The question arises why for some it works and for others, it does not and has to do some workarounds.

Thanks,

Re: DAB deploy missing packages

-werners- — Tue, 22 Oct 2024 11:29:06 GMT

it could be depending on the template you used when creating the package.
DAB is still rough around the edges.

Re: DAB deploy missing packages

Przemk00 — Tue, 22 Oct 2024 11:50:18 GMT

Is there anything specific I should look in the template?

Re: DAB deploy missing packages

-werners- — Tue, 22 Oct 2024 14:19:18 GMT

In your case it would be what is in the .gitignore file.
Somehow your files were ignored, and if you did not edit the .gitignore, it was probably copied from a template (or existing dab).

Re: DAB deploy missing packages

saurabh18cs — Tue, 22 Oct 2024 14:21:03 GMT

Production native approach is to maintain your artifacts in central antifactory and use this location in your .yml for libraries. This way you ensure your antifactory are properly versioned , governed and maintaining some sort of standards. I hope with this approach you can reduce your errors of missing packages/whl etc.

e.g.

libraries:

- pypi:

package: requests==2.31.0

- pypi:

package: azure-eventhub==5.10.1

- pypi:

package: azure-identity==1.12.0

- pypi:

package: yourpackage==0.0.0

repo: https://YOURPACKAGE_ARTIFACT_REPO:YOURPACKAGE_ARTIFACT_TOKEN@pkgs.dev.azure.com/asml/c3d5....................../_packaging/FEED_NAME/pypi/simple/

Re: DAB deploy missing packages

saurabh18cs — Tue, 22 Oct 2024 14:23:28 GMT

e.g.

libraries:

- pypi:

package: requests==2.31.0

- pypi:

package: yourpackage==0.0.0

repo: https://YOURPACKAGE_ARTIFACT_REPO:YOURPACKAGE_ARTIFACT_TOKEN@pkgs.dev.azure.com/asml/c3d5....................../_packaging/FEED_NAME/pypi/simple/