1. The error in scenario 3 is likely due to the fact that the service principal is not an owner of the DLT pipeline that creates the materialized views. Even though the job is running on a shared cluster, the service principal still needs to be an owner of the pipeline to run the "describe table extended" command on materialized views.
2. The error messages indicate that the user running the command does not have the necessary permissions to access the materialized views. The error in scenario 3 is likely due to the fact that the service principal is not an owner of the DLT pipeline that creates the materialized views.
3. One solution for this problem is to use the "information_schema" to exclude materialized views from the loop. You can use the "information_schema.views" table to get a list of all views and materialized views in the database, and then exclude them from the loop. Here's an example of how you can do this:

db_tables = []
for database in spark.sql("show databases").collect():
  # Skip the 'information_schema' database
  if database['databaseName'].lower() != 'information_schema':
    for table in spark.sql(f"SHOW TABLES IN {database['databaseName']}").collect():
      # Check if the table is temporary
      if not table['isTemporary']:
        # Use INFORMATION_SCHEMA.VIEWS to check if it's a view or a table
        desc_table = spark.sql(f"SELECT * FROM {database['databaseName']}.information_schema.views WHERE table_name = '{table['tableName']}'").collect()
       
        if len(desc_table) > 0 and desc_table[0]['table_type'] in ('VIEW', 'MATERIALIZED_VIEW'):
          continue # Skip if it's a view or materialized view
        else:
          db_tables.append([database['databaseName'],table['tableName']])