Arguments parsing in Databricks python jobs

Klusener — Sun, 26 Nov 2023 09:55:48 GMT

On Databricks created a job task with task type as Python script from s3. However, when arguments are passed via Parameters option, running into unrecognized arguments' error.

Code in s3 file:

import argparse

def parse_arguments():
    parser = argparse.ArgumentParser(description='Parse and print command line arguments')
    parser.add_argument('--first_argument', action='store_true', help='first argument')
    parser.add_argument('--second_argument', action='store_true', help='second argument')

    args = parser.parse_args()

    return args

def main():
    args = parse_arguments()

    print('--first_argument', args.first_argument)
    print('--second_argument', args.second_argument)

if __name__ == "__main__":
    main()

Approach-1: parameters passed

["--first_argument","val1","--second_argument","val2"]

Error: tmpm3.py: error: unrecognized arguments: val1 val2

Approach-2: parameters passed

["--first_argument val1","--second_argument val2"]

Error: tmpshby3l: error: unrecognized arguments: --first_argument val1 --second_argument val2

topic Arguments parsing in Databricks python jobs in Get Started Discussions

Arguments parsing in Databricks python jobs