dbx - run unit test error (java.lang.NoSuchMethodError)

aka1 — Tue, 22 Nov 2022 18:23:20 GMT

I am setting up dbx for the fist time on Windows 10.

Strictly following https://dbx.readthedocs.io/en/latest/guides/python/python_quickstart/

openjdk is installed

conda install -c conda-forge openjdk=11.0.15

winutils.exe for Hadoop 3 is downloaded, path is set up

Local Dependencies:

Spark 3.3.1:

dbx version

I am stuck on the final step (unit tests run):

pytest tests/unit

Error: java.lang.NoSuchMethodError: 'void org.apache.spark.sql.execution.datasources.parquet.ParquetSchemaConverter$.checkFieldName(java.lang.String)'

    if is_error(answer)[0]:
            if len(answer) > 1:
                type = answer[1]
                value = OUTPUT_CONVERTER[type](answer[2:], gateway_client)
                if answer[1] == REFERENCE_TYPE:
>                   raise Py4JJavaError(
                        "An error occurred while calling {0}{1}{2}.\n".
                        format(target_id, ".", name), value)
E                   py4j.protocol.Py4JJavaError: An error occurred while calling o59.saveAsTable.
E                   : java.lang.NoSuchMethodError: 'void org.apache.spark.sql.execution.datasources.parquet.ParquetSchemaConverter$.checkFieldName(java.lang.String)'
E                       at org.apache.spark.sql.delta.schema.SchemaUtils$.$anonfun$checkFieldNames$1(SchemaUtils.scala:890)

Could you please help me to solve the issue ?

Re: dbx - run unit test error (java.lang.NoSuchMethodError)

Aviral-Bhardwaj — Sun, 18 Dec 2022 07:28:09 GMT

this seems code issue only

topic Re: dbx - run unit test error (java.lang.NoSuchMethodError) in Data Engineering

dbx - run unit test error (java.lang.NoSuchMethodError)

Re: dbx - run unit test error (java.lang.NoSuchMethodError)