https://community.databricks.com/t5/data-engineering/error-using-spark-catalog-droptempview/m-p/13589#M8247 <P>&nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; spark.sql("DROP TABLE "+prefix_updates)</P><P>&nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; spark.sql("DROP TABLE "+prefix_main)</P><P></P><P>Fixed it for me.</P> Thu, 05 Jan 2023 03:49:03 GMT aicd_de 2023-01-05T03:49:03Z https://community.databricks.com/t5/data-engineering/error-using-spark-catalog-droptempview/m-p/13588#M8246 <P>I have a set of Spark Dataframes that I convert into Temp Views to run Spark SQL with. Then, I delete them after my logic/use is complete. The delete step throws an odd error that I am not sure how to fix. Looking for some tips on fixing it. As a note, the cluster is on Unity as a Shared Cluster.</P><P></P><PRE><CODE>df.createOrReplaceTempView(prefix_updates) target.toDF().createOrReplaceTempView(prefix_main) &nbsp; sql_begin = 'SELECT '+prefix_updates+'.* FROM '+prefix_updates+' INNER JOIN '+prefix_main+' ON ' &nbsp; merged_inserts = spark.sql(sql_begin+merge_match_ins+" WHERE "+prefix_updates+".ReplicationUTCDateTime &gt; "+prefix_main+".ReplicationUTCDateTime") merged_inserts.write.format("delta").mode("append").saveAsTable(catalog+'.'+schema+'.'+table_name) &nbsp; spark.catalog.dropTempView(prefix_updates) spark.catalog.dropTempView(prefix_main)</CODE></PRE><P>The code before spark.catalog.dropTempView works just fine without error. But when adding in the View delete statements I get this error now:</P><P></P><P>Py4JError: An error occurred while calling o1620.dropTempView. Trace:</P><P>py4j.security.Py4JSecurityException: Method public boolean org.apache.spark.sql.internal.CatalogImpl.dropTempView(java.lang.String) is not whitelisted on class class org.apache.spark.sql.internal.CatalogImpl</P><P> at py4j.security.WhitelistingPy4JSecurityManager.checkCall(WhitelistingPy4JSecurityManager.java:473)</P><P> at py4j.Gateway.invoke(Gateway.java:305)</P><P> at py4j.commands.AbstractCommand.invokeMethod(AbstractCommand.java:132)</P><P> at py4j.commands.CallCommand.execute(CallCommand.java:79)</P><P> at py4j.ClientServerConnection.waitForCommands(ClientServerConnection.java:195)</P><P> at py4j.ClientServerConnection.run(ClientServerConnection.java:115)</P><P> at java.lang.Thread.run(Thread.java:750)</P> Thu, 05 Jan 2023 01:54:52 GMT https://community.databricks.com/t5/data-engineering/error-using-spark-catalog-droptempview/m-p/13588#M8246 aicd_de 2023-01-05T01:54:52Z https://community.databricks.com/t5/data-engineering/error-using-spark-catalog-droptempview/m-p/13589#M8247 <P>&nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; spark.sql("DROP TABLE "+prefix_updates)</P><P>&nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; spark.sql("DROP TABLE "+prefix_main)</P><P></P><P>Fixed it for me.</P> Thu, 05 Jan 2023 03:49:03 GMT https://community.databricks.com/t5/data-engineering/error-using-spark-catalog-droptempview/m-p/13589#M8247 aicd_de 2023-01-05T03:49:03Z