Hello everyone,

I'm experiencing a very strange issue with temporary views in Databricks 15.4 LTS that did not occur in 13.3. I have a workflow where I create a temporary view, run a query against it, and then drop it using a DROP VIEW IF EXISTS command. The issue is as follows:

Issue Description

• I create a temporary view using a dynamically generated name:

temp_view_name = f"{self.input_data}_{uuid4().hex}" df.createOrReplaceTempView(temp_view_name)

• I then run a query against the view (which works fine), and afterwards, I attempt to drop the view using an f-string:

   

self.spark.sql(f"DROP VIEW IF EXISTS {temp_view_name}")

• However, this DROP statement fails with an error:

   

[TABLE_OR_VIEW_NOT_FOUND] The table or view `input_df_<uuid>` cannot be found. Verify the spelling and correctness of the schema and catalog. ...

• Curiously, if I hardcode the entire DROP statement as a literal string:

self.spark.sql('''
DROP VIEW IF EXISTS input_df_5bf8576e20da4c49a986ed81428ea839
''')

the DROP works as expected.

What I've Tried

• Quoting the Identifier:
  I tried wrapping the temporary view name in backticks:

  python

   

self.spark.sql(f"DROP VIEW IF EXISTS `{temp_view_name}`")

but the error persists.

• Using the Spark Catalog API:
  I also attempted:

self.spark.catalog.dropTempView(temp_view_name)

which yields the same TABLE_OR_VIEW_NOT_FOUND error.

• Verifying the View's Existence:
  Right after creating the view, I list all session tables with:

print([t.name for t in self.spark.catalog.listTables()])

and the temporary view is present in the list.

Environment Details

• Working Runtime: Databricks 13.3 LTS (no issues with DROP VIEW IF EXISTS using dynamic names).

• Problematic Runtime: Databricks 15.4 LTS.

• View Creation: The temporary view is created successfully and is visible when listing tables.

• DROP Behavior: When dropping using a literal SQL string, it works; when dropping using a dynamic f-string (or the catalog API), it fails with TABLE_OR_VIEW_NOT_FOUND.

Questions

1. Has anyone encountered this discrepancy in 15.4 LTS regarding dynamic temporary view drop behavior?

2. Could this be a bug or a change in how the session catalog resolves temporary views in 15.4 LTS?

3. Are there any recommended workarounds or settings to ensure that DROP VIEW IF EXISTS works with dynamically generated temporary view names?