topic Re: Databricks Job: Package Name and EntryPoint parameters for the Python Wheel file in Data Engineering

Databricks Job: Package Name and EntryPoint parameters for the Python Wheel file

sandeep91 — Tue, 01 Mar 2022 16:53:30 GMT

I have created Python wheel file with simple file structure and uploaded into cluster library and was able to run the packages in Notebook but, when I am trying to create a Job using python wheel and provide the package name and run the task it fails with Wheel Name not found or package not found.

Can you help us with exact parameters need to be pass into package Name and entrypoint. I followed doc but with no success.

File structure:

| - PythonWheel

| - src

| - __init__.py

| - test1.py

| - test2.py

| - setup.py

Setup.py:

from setuptools import find_packages, setup

setup(

name="testbronze",

packages = find_packages(),

setup_requires=["wheel"],

description="demo",

version="0.0.1",

include_package_data=True,

)

Job PackageName:

testbronze (Note: also tried with "src")

Re: Databricks Job: Package Name and EntryPoint parameters for the Python Wheel file

Hubert-Dudek — Tue, 01 Mar 2022 19:27:48 GMT

Advancing analytics explain wheel in that highly recommended video https://www.youtube.com/watch?v=nN-NPnfJLNY That video also explaining how to use files in repos which is more better solution than wheel package (as wheel have to be installed on server every time, files in repos can just stay in your git repo)

Re: Databricks Job: Package Name and EntryPoint parameters for the Python Wheel file

sandeep91 — Tue, 01 Mar 2022 22:11:46 GMT

Thank you for your reply @Hubert Dudek

But, based on my use case I cannot use Notebook to run the Job.

We need to orchestrate directly through task using python wheel option.

I found many docs pointing to run through Notebook but no documents refer directly to trigger task using wheel file.

Re: Databricks Job: Package Name and EntryPoint parameters for the Python Wheel file

sandeep91 — Thu, 03 Mar 2022 19:21:58 GMT

I have resolved the Issue.

Problem was in Setup.py

In the name parameter I need to provide the proper structure of my package

Note: If your package structure is src.bronze you need to name it exactly.

setup(

name="src",

packages = ['.src'],

description="demo",

version="0.1",

author="Sandeep",

long_description=long_description,

long_description_content_type="text/markdown",

license='LICENSE.txt',

include_package_data=True,

)

And from the Databricks UI:

PackageName: src

Entry_Point : print_name (Function)

Re: Databricks Job: Package Name and EntryPoint parameters for the Python Wheel file

Anonymous — Sun, 06 Mar 2022 22:13:04 GMT

@Sandeep Toopran - Thank you for letting us know how you solved the problem!

Re: Databricks Job: Package Name and EntryPoint parameters for the Python Wheel file

AndréSalvati — Wed, 06 Mar 2024 17:00:50 GMT

There you can see a complete template project with (the new!!!) Databricks Asset Bundles tool and a python wheel task. Please, follow the instructions for deployment.

https://github.com/andre-salvati/databricks-template