Arguments parsing in Databricks python jobs
Options
- Mark as New
- Bookmark
- Subscribe
- Mute
- Subscribe to RSS Feed
- Permalink
- Report Inappropriate Content
11-26-2023 01:55 AM
On Databricks created a job task with task type as Python script from s3. However, when arguments are passed via Parameters option, running into unrecognized arguments' error.
Code in s3 file:
import argparse def parse_arguments(): parser = argparse.ArgumentParser(description='Parse and print command line arguments') parser.add_argument('--first_argument', action='store_true', help='first argument') parser.add_argument('--second_argument', action='store_true', help='second argument') args = parser.parse_args() return args def main(): args = parse_arguments() print('--first_argument', args.first_argument) print('--second_argument', args.second_argument) if __name__ == "__main__": main()
Approach-1: parameters passed
["--first_argument","val1","--second_argument","val2"]
Error: tmpm3.py: error: unrecognized arguments: val1 val2
Approach-2: parameters passed
["--first_argument val1","--second_argument val2"]
Error: tmpshby3l: error: unrecognized arguments: --first_argument val1 --second_argument val2