This is not a question, this is just the solution to a problem we encountered in case someone from the community finds it useful.
Recently we encountered an issue, where our users' jobs started failing out of nowhere on the following command, with the error: java.util.NoSuchElementException: None.get
spark.sql("SHOW TABLES in catalog.bucket")
Solution: Use DBR version 11.3 and above.
Root Cause
For us the reason for the failures in 10.4 DBR and below was that in the target catalog and bucket, there were some new types of tables which were used for LLM projects (vector indexes of type Delta Sync). If we would try to "DESCRIBE" that vector index with DBR 10.4, we would get the below error. This was also the cause for the error mentioned above.
