topic Re: Using Databricks asset bundles with typer instead of argparse in Data Engineering

Using Databricks asset bundles with typer instead of argparse

VicS — Thu, 14 Nov 2024 10:32:44 GMT

I want to use Databricks asset bundles - I'd like to use `typer` as a CLI tool, but I have only been able to set it up with `argparse`. Argparse seems to be able to retrieve the arguments from the databricks task, but not typer.

I specified two entrypoints in my pyproject.toml

[tool.poetry.scripts] mypackage_ep_typer = "my_package.entrypoint_typer:main" mypackage_ep_argparse = "my_package.entrypoint_argparse:entrypoint_generic"

I'm able to use the asset bundles with an entrypoint to my ELTL applications with argparse as follows in the entrypoint_argparse.py:

def entrypoint_generic(): """Execute the application.""" logger.info("Executing 'argparse' entrypoint...") parser = argparse.ArgumentParser(description="My module.") parser.add_argument( "--applicationname", help="The name of the application to execute.", type=str, required=True, dest="applicationname", ) args = parser.parse_args() logger.info(f"{args.applicationname=}")

This one deploys and runs without any issues in the asset bundle's workflow task.

However, if I try the same with typer in entrypoint_typer.py, I cannot get it to work:

def main( applicationname: Annotated[str, typer.Option(help="Application to execute.")] ): """Execute the application.""" logger.info("Executing 'typer' entrypoint...") logger.info(f"{applicationname=}")

I can run typer locally:

> poetry run python -m typer .\entrypoint_typer.py run --applicationname MYAPPNAME 2024-11-06 xx:xx:xx - root - INFO - Executing 'typer' entrypoint... 2024-11-06 xx:xx:xx - root - INFO - applicationname='MYAPPNAME'

But when I try to deploy and run my asset bundle, I get this error when the workflow task tries to start:

TypeError: main() missing 1 required positional argument: 'applicationname'

But I can see the parameter in the UI...

Re: Using Databricks asset bundles with typer instead of argparse

cgrant — Fri, 22 Nov 2024 03:34:25 GMT

When you used named_parameters in Databricks Workflows' python_wheel_task type, the arguments will be available to your program in form of --key=value, literally. Argparse can parse this, but can Typer? If not, you can switch from named_parameters to parameters where you have a bit more flexibility in how the parameters are presented, like space between key and value instead of equals sign.

Re: Using Databricks asset bundles with typer instead of argparse

VicS — Fri, 22 Nov 2024 08:13:22 GMT

Typer can handle this syntax as well, I just tested it locally by passing the value for application-name in various formats:

Test 1

PS C:\my_repo> poetry run python -m typer .\entrypoint_typer.py run --application-name=182 2024-11-22 09:09:44,223 - root - INFO - application_name='182'

Test 2

PS C:\my_repo> poetry run python -m typer .\entrypoint_typer.py run --application-name 182 2024-11-22 09:10:21,266 - root - INFO - application_name='182'

Still unsure why it doesn't work as part of the asset bundle though. It's not a critical issue since we can use argparse instead, but we still would love to know why one works while the other doesn't. Maybe it does come down to the implementation of typer, some aspect might be different when running it in Databricks.

Re: Using Databricks asset bundles with typer instead of argparse

NandiniN — Fri, 10 Jan 2025 07:11:37 GMT

If switching to parameters does not resolve the issue, you might need to further debug by adding logging statements in your typer entry point to see how the parameters are being received.