Databricks Community

aka1 · ‎11-22-2022

I am setting up dbx for the fist time on Windows 10.

Strictly following https://dbx.readthedocs.io/en/latest/guides/python/python_quickstart/

openjdk is installed

conda install -c conda-forge openjdk=11.0.15

winutils.exe for Hadoop 3 is downloaded, path is set up

Local Dependencies:

Spark 3.3.1:

dbx version

I am stuck on the final step (unit tests run):

pytest tests/unit

Error: java.lang.NoSuchMethodError: 'void org.apache.spark.sql.execution.datasources.parquet.ParquetSchemaConverter$.checkFieldName(java.lang.String)'

    if is_error(answer)[0]:
            if len(answer) > 1:
                type = answer[1]
                value = OUTPUT_CONVERTER[type](answer[2:], gateway_client)
                if answer[1] == REFERENCE_TYPE:
>                   raise Py4JJavaError(
                        "An error occurred while calling {0}{1}{2}.\n".
                        format(target_id, ".", name), value)
E                   py4j.protocol.Py4JJavaError: An error occurred while calling o59.saveAsTable.
E                   : java.lang.NoSuchMethodError: 'void org.apache.spark.sql.execution.datasources.parquet.ParquetSchemaConverter$.checkFieldName(java.lang.String)'
E                       at org.apache.spark.sql.delta.schema.SchemaUtils$.$anonfun$checkFieldNames$1(SchemaUtils.scala:890)

Could you please help me to solve the issue ?