I am getting below error some time run my databricks notebook from ADF, 

If the executor node is one then it works fine, if it increases 2 or more some times its failing on same data.

Cluster Detail : Standard_F4s_v2 · Workers: Standard_F4s_v2 · 1-8 workers · 11.2 (includes Apache Spark 3.3.0, Scala 2.12)

File "/databricks/python/lib/python3.9/site-packages/Levenshtein/__init__.py", line 343 in opcodes

 File "/databricks/python/lib/python3.9/site-packages/fuzzywuzzy/StringMatcher.py", line 45 in get_opcodes

 File "/databricks/python/lib/python3.9/site-packages/fuzzywuzzy/StringMatcher.py", line 58 in get_matching_blocks

 File "/databricks/python/lib/python3.9/site-packages/fuzzywuzzy/fuzz.py", line 47 in partial_ratio

 File "/databricks/python/lib/python3.9/site-packages/fuzzywuzzy/utils.py", line 47 in decorator

 File "/databricks/python/lib/python3.9/site-packages/fuzzywuzzy/utils.py", line 29 in decorator

 File "/databricks/python/lib/python3.9/site-packages/fuzzywuzzy/utils.py", line 38 in decorator

 File "/databricks/python/lib/python3.9/site-packages/my_package/my_function.py", line 30 in scrap_url

 File "/databricks/python/lib/python3.9/site-packages/my_package/my_function.py", line 124 in my_function

 File "<command-1514877556254536>", line 20 in my_function

 File "<command-1514877556254534>", line 7 in my_function_01

 File "/databricks/spark/python/pyspark/util.py", line 84 in wrapper

 File "/databricks/spark/python/pyspark/worker.py", line 130 in <lambda>

 File "/databricks/spark/python/pyspark/worker.py", line 591 in mapper

 File "/databricks/spark/python/pyspark/sql/pandas/serializers.py", line 384 in init_stream_yield_batches

 File "/databricks/spark/python/pyspark/sql/pandas/serializers.py", line 91 in dump_stream

 File "/databricks/spark/python/pyspark/sql/pandas/serializers.py", line 391 in dump_stream

 File "/databricks/spark/python/pyspark/worker.py", line 885 in process

 File "/databricks/spark/python/pyspark/worker.py", line 893 in main

 File "/databricks/spark/python/pyspark/daemon.py", line 79 in worker

 File "/databricks/spark/python/pyspark/daemon.py", line 204 in manager

 File "/databricks/spark/python/pyspark/daemon.py", line 229 in <module>

 File "/usr/lib/python3.9/runpy.py", line 87 in _run_code

 File "/usr/lib/python3.9/runpy.py", line 197 in _run_module_as_main

Any one please help me on this . is that time lag issue with a task?

Some times its working and sometimes dont.

If didn't add package "Levenshtein" 3 tasks are taking 2 h to complete.