topic OSError: [Errno 7] Argument list too long in Data Engineering

OSError: [Errno 7] Argument list too long

MUA — Wed, 16 Aug 2023 19:23:48 GMT

Getting this error in Databricks and don't know how to solve

OSError: [Errno 7] Argument list too long: '/dbfs/databricks/aaecz/dev/w000aaecz/etl-framework-adb/0.4.31-20230503.131701-1/etl_libraries/utils/datadog/restart_datadog.sh'

if anyone can help

Re: OSError: [Errno 7] Argument list too long

SparkJun — Fri, 01 Sep 2023 21:39:46 GMT

Hi MUA, can you please firstly validate this DBFS path by using %fs ls bash command to list out the objects in this path? Then, the error OSError: [Errno 7] Argument list too long occurs when the number of command line arguments or the total size of the command line arguments is larger than the system's limit. This is a system-level error, not specific to any particular programming language or application by Spark. If the arguments can't be reduced, consider storing these arguments in a file and modify your script or command to read from this file instead of taking arguments from the command line.

args = json.loads(json_file_path)
print(args)

Re: OSError: [Errno 7] Argument list too long

jose_gonzalez — Fri, 08 Sep 2023 22:55:39 GMT

@MUA

Just a friendly follow-up. Did any of the responses help you to resolve your question? if it did, please mark it as best. Otherwise, please let us know if you still need help.