Hi @Debayan Mukherjee ,
there is the whole output
>>> spark = get_spark()
:: loading settings :: url = jar:file:/usr/local/lib/python3.8/dist-packages/pyspark/jars/ivy-2.5.0.jar!/org/apache/ivy/core/settings/ivysettings.xml
Ivy Default Cache set to: /root/.ivy2/cache
The jars for the packages stored in: /root/.ivy2/jars
io.delta#delta-core_2.12 added as a dependency
com.databricks#spark-xml_2.12 added as a dependency
org.postgresql#postgresql added as a dependency
:: resolving dependencies :: org.apache.spark#spark-submit-parent-1300bbeb-d826-46b8-a3f0-2502ebf354af;1.0
confs: [default]
found io.delta#delta-core_2.12;1.1.0 in central
found org.antlr#antlr4-runtime;4.8 in central
found org.codehaus.jackson#jackson-core-asl;1.9.13 in central
found com.databricks#spark-xml_2.12;0.14.0 in central
found commons-io#commons-io;2.8.0 in central
found org.glassfish.jaxb#txw2;2.3.4 in central
found org.apache.ws.xmlschema#xmlschema-core;2.2.5 in central
found org.postgresql#postgresql;42.2.23 in central
found org.checkerframework#checker-qual;3.5.0 in central
:: resolution report :: resolve 418ms :: artifacts dl 19ms
:: modules in use:
com.databricks#spark-xml_2.12;0.14.0 from central in [default]
commons-io#commons-io;2.8.0 from central in [default]
io.delta#delta-core_2.12;1.1.0 from central in [default]
org.antlr#antlr4-runtime;4.8 from central in [default]
org.apache.ws.xmlschema#xmlschema-core;2.2.5 from central in [default]
org.checkerframework#checker-qual;3.5.0 from central in [default]
org.codehaus.jackson#jackson-core-asl;1.9.13 from central in [default]
org.glassfish.jaxb#txw2;2.3.4 from central in [default]
org.postgresql#postgresql;42.2.23 from central in [default]
---------------------------------------------------------------------
| | modules || artifacts |
| conf | number| search|dwnlded|evicted|| number|dwnlded|
---------------------------------------------------------------------
| default | 9 | 0 | 0 | 0 || 9 | 0 |
---------------------------------------------------------------------
:: retrieving :: org.apache.spark#spark-submit-parent-1300bbeb-d826-46b8-a3f0-2502ebf354af
confs: [default]
0 artifacts copied, 9 already retrieved (0kB/19ms)
22/12/20 09:45:23 WARN NativeCodeLoader: Unable to load native-hadoop library for your platform... using builtin-java classes where applicable
Setting default log level to "WARN".
To adjust logging level use sc.setLogLevel(newLevel). For SparkR, use setLogLevel(newLevel).
>>> spark.sql("select '1'")
Traceback (most recent call last):
File "<stdin>", line 1, in <module>
File "/usr/local/lib/python3.8/dist-packages/pyspark/sql/session.py", line 1034, in sql
return DataFrame(self._jsparkSession.sql(sqlQuery), self)
File "/usr/local/lib/python3.8/dist-packages/py4j/java_gateway.py", line 1321, in __call__
return_value = get_return_value(
File "/usr/local/lib/python3.8/dist-packages/pyspark/sql/utils.py", line 190, in deco
return f(*a, **kw)
File "/usr/local/lib/python3.8/dist-packages/py4j/protocol.py", line 326, in get_return_value
raise Py4JJavaError(
py4j.protocol.Py4JJavaError: An error occurred while calling o47.sql.
: java.lang.NoSuchMethodError: org.apache.spark.sql.catalyst.parser.ParserUtils$.withOrigin(Lorg/antlr/v4/runtime/ParserRuleContext;Lscala/Function0;)Ljava/lang/Object;
at io.delta.sql.parser.DeltaSqlAstBuilder.visitSingleStatement(DeltaSqlParser.scala:188)
at io.delta.sql.parser.DeltaSqlAstBuilder.visitSingleStatement(DeltaSqlParser.scala:143)
at io.delta.sql.parser.DeltaSqlBaseParser$SingleStatementContext.accept(DeltaSqlBaseParser.java:160)
at org.antlr.v4.runtime.tree.AbstractParseTreeVisitor.visit(AbstractParseTreeVisitor.java:18)
at io.delta.sql.parser.DeltaSqlParser.$anonfun$parsePlan$1(DeltaSqlParser.scala:71)
at io.delta.sql.parser.DeltaSqlParser.parse(DeltaSqlParser.scala:100)
at io.delta.sql.parser.DeltaSqlParser.parsePlan(DeltaSqlParser.scala:70)
at org.apache.spark.sql.SparkSession.$anonfun$sql$2(SparkSession.scala:620)
at org.apache.spark.sql.catalyst.QueryPlanningTracker.measurePhase(QueryPlanningTracker.scala:111)
at org.apache.spark.sql.SparkSession.$anonfun$sql$1(SparkSession.scala:620)
at org.apache.spark.sql.SparkSession.withActive(SparkSession.scala:779)
at org.apache.spark.sql.SparkSession.sql(SparkSession.scala:617)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:498)
at py4j.reflection.MethodInvoker.invoke(MethodInvoker.java:244)
at py4j.reflection.ReflectionEngine.invoke(ReflectionEngine.java:357)
at py4j.Gateway.invoke(Gateway.java:282)
at py4j.commands.AbstractCommand.invokeMethod(AbstractCommand.java:132)
at py4j.commands.CallCommand.execute(CallCommand.java:79)
at py4j.ClientServerConnection.waitForCommands(ClientServerConnection.java:182)
at py4j.ClientServerConnection.run(ClientServerConnection.java:106)
at java.lang.Thread.run(Thread.java:750)
more precisely，I ran into the issue with Databricks runtime 11.3, which runs on spark 3.3.0. I tried to downgrade spark to 3.2.1. Despite the incompatble error, the code worked.
