Hi,  I am using Databricks and want to upgrade to Databricks runtime version 11.3 LTS which uses Spark 3.3 now. 

Current system enviroment:

• Operating System: Ubuntu 20.04.4 LTS
• Java: Zulu 8.56.0.21-CA-linux64
• Python: 3.8.10
• Delta Lake: 1.1.0

Target system enviroment:

• Operating System: Ubuntu 20.04.5 LTS
• Java: Zulu 8.56.0.21-CA-linux64
• Python: 3.9.5
• Delta Lake: 2.1.0

After Upgrade, when I try to use the databricks-connector with Spark 3.3 my Spark jobs crash with the following stack trace:

py4j.protocol.Py4JJavaError: An error occurred while calling o47.sql.
: java.lang.NoSuchMethodError: org.apache.spark.sql.catalyst.parser.ParserUtils$.withOrigin(Lorg/antlr/v4/runtime/ParserRuleContext;Lscala/Function0;)Ljava/lang/Object;
        at io.delta.sql.parser.DeltaSqlAstBuilder.visitSingleStatement(DeltaSqlParser.scala:188)
        at io.delta.sql.parser.DeltaSqlAstBuilder.visitSingleStatement(DeltaSqlParser.scala:143)
        at io.delta.sql.parser.DeltaSqlBaseParser$SingleStatementContext.accept(DeltaSqlBaseParser.java:160)
        at org.antlr.v4.runtime.tree.AbstractParseTreeVisitor.visit(AbstractParseTreeVisitor.java:18)
        at io.delta.sql.parser.DeltaSqlParser.$anonfun$parsePlan$1(DeltaSqlParser.scala:71)
        at io.delta.sql.parser.DeltaSqlParser.parse(DeltaSqlParser.scala:100)
        at io.delta.sql.parser.DeltaSqlParser.parsePlan(DeltaSqlParser.scala:70)
        at org.apache.spark.sql.SparkSession.$anonfun$sql$2(SparkSession.scala:620)
        at org.apache.spark.sql.catalyst.QueryPlanningTracker.measurePhase(QueryPlanningTracker.scala:111)
        at org.apache.spark.sql.SparkSession.$anonfun$sql$1(SparkSession.scala:620)
        at org.apache.spark.sql.SparkSession.withActive(SparkSession.scala:779)
        at org.apache.spark.sql.SparkSession.sql(SparkSession.scala:617)
        at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
        at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
        at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
        at java.lang.reflect.Method.invoke(Method.java:498)
        at py4j.reflection.MethodInvoker.invoke(MethodInvoker.java:244)
        at py4j.reflection.ReflectionEngine.invoke(ReflectionEngine.java:357)
        at py4j.Gateway.invoke(Gateway.java:282)
        at py4j.commands.AbstractCommand.invokeMethod(AbstractCommand.java:132)
        at py4j.commands.CallCommand.execute(CallCommand.java:79)
        at py4j.ClientServerConnection.waitForCommands(ClientServerConnection.java:182)
        at py4j.ClientServerConnection.run(ClientServerConnection.java:106)
        at java.lang.Thread.run(Thread.java:750)

After investigation, i found this caued by Spark 3.3.x. Becasue the code worked perfectly with Spark 3.2.x

It blocking our upgrade from 10.4 to 11.3 runtime on Databricks

Can you please guide me regarding the same.